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SUMMARY 


The  Advanced  On-the-job  Training  System  (AOTS)  was  an  Air  Staff-directed, 
AFHRL-developed  prototype  which  designed,  developed,  and  tested  a  proof-of- 
concept  prototype  AOTS  within  the  operational  environment  of  selected  work 
centers  at  Bergstrom  AFB,  Texas,  and  Ellington  AN6B,  Texas,  from  August  1985 
through  31  July  1989.  The  Master  Test  Plan  (MTP)  describes  the  overall 
strategy  that  was  followed  during  the  test  and  evaluation  of  the  Advanced 
On-the-job  Training  (AOTS).  The  Master  Test  Plan  is  synonymous  with  program 
evaluation  and  consists  of  a  plan  to  evaluate  AOTS  with  regard  to  assessment 
of  the  four  critical  issues  of  system  compliance,  acceptance,  performance  and 
suitability.  Within  the  MTP,  these  critical  issues  are  assessed  at  sub¬ 


component,  component,  and  subsystem  levels. 


PREFACE 


This  paper  was  developed  by  Douglas  Aircraft  Company,  the  AOTS  develop¬ 
ment  contractor,  under  Government  Contract  Number  F33615-C-84-0059.  The 
AFHRL  Work  Unit  Number  for  the  project  is  2557-00-02.  The  primary  office  of 
responsibility  for  management  of  the  work  unit  is  the  Air  Force  Human 
Resources  Laboratory,  Training  Systems  Division,  and  the  Air  Force  AOTS 
manager  is  Major  Jack  Blackhurst. 
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1.0  DESCRIPTION 


1 . 1  Mission 

1.1.1  Introduction.  The  primary  mission  of  the  Advanced  On-the- 
job  Training  System  (AOTS)  is  to  supply  the  Air  Force  with  airmen 
who  are  position  qualified  within  operational  duty  positions.  To 
meet  the  mission  objective,  AOTS  must  focus  on-the-job  training 
(OJT)  on  job  task  proficiency  by  specifying  and  defining  the 
tasks  required  in  a  given  duty  position  and  by  adequately  defin¬ 
ing  the  training  required  to  become  fully  position  qualified. 
AOTS  must  provide  workable  and  cost  effective  methods  for 
evaluating  task  performance  in  an  operational  setting.  AOTS  must 
also  ease  the  inherent  difficulties  of  using  operational  equip¬ 
ment  for  OJT.  It  must  address  the  problem  of  the  limited 
availability  of  technically  qualified  personnel  who  can  act  as 
trainers  and  the  limited  training  skills  of  those  personnel. 
AOTS  must  ease  the  paperwork  burden  associated  with  scheduling 
both  expected  and  unique  OJT  opportunities,  tracking  trainee 
progress,  evaluating  associated  knowledge  test  and  task  perfor¬ 
mance,  and  managing  the  flow  of  OJT  data.  It  must  ultimately  in¬ 
crease  individual  and  unit  productivity  and  readiness. 

1-1.2  Operational  Concept.  The  prototype  AOTS  will  demonstrate 
the  management  and  evaluation  functions  for  OJT  in  five  Air  Force 
Specialties  (AFSs)  within  eight  regular  Air  Force,  five  Reserve, 
and  six  Air  National  Guard  (ANG)  workcenters.  AOTS  will  provide 
the  capabilities  and  methodologies  to: 

a.  identify  and  define  task  performance  and  training 
requirements 

b.  schedule  training  and  identify  evaluation  resource 
requirements 
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c.  track  individual  airman  training  accomplishments 

d.  evaluate  individual  airman  task  knowledge  and 
performance 

e.  track,  evaluate,  and  report  the  AOTS  program  effective¬ 
ness 

f.  report  airman  training,  evaluation  status,  and 
evaluation  results  to  appropriate  Air  Force  managers. 

AOTS  will  primarily  be  a  computer  based  system.  Editors, 
data  bases,  and  system  utilities  will  be  made  available,  either 
in  whole  or  in  part,  to  various  Air  Force  users.  These  users 
will  include:  developers  and  managers  of  AOTS  information  such 
as  ISD  personnel  and  AOTS  operators;  end  users  of  AOTS  products 
such  as  trainees,  trainers,  and  evaluators;  and  Air  Force 
managers  such  as  supervisors,  base  level  commanders.  Air  Staff, 
etc.  AOTS  design  incorporates  the  concepts  of  user  friendliness, 
flexibility,  and  efficiency.  The  system  support  concept  incor¬ 
porates  a  formalized  logistics  plan  for  hardware,  software,  and 
personnel . 

1.2  Key  Functional  Characteristics 

1.2.1  AOTS  Management  Subsystem.  The  Management  Subsystem 
provides  computer  based  capabilities  and  methodologies  to:  iden¬ 
tify  Air  Force  Specialty  and  other  training  required  to  achieve 
position  qualification  within  an  assigned  operational  duty 
position;  manage  and  record  airman  training  progress  toward  task 
proficiency  and  position  qualification  within  an  operational  duty 
position;  schedule  training  and  evaluations;  and  identify  base 
level  training  and  evaluation  resource  requirements. 

1.2.2  AOTS  Evaluation  Subsystem.  The  Evaluation  Subsystem 
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Drovides  computer  based  capabilities  and  methodologies  to: 
nanage  evaluation  instrumentation  including  the  planning  and 
development  of  objectives  and  tests;  evaluate  performance  of  air¬ 
men  in  both  objective-based  knowledge  tests  and  performance 
evaluations;  provide  standardized  quality  control  evaluation  pro¬ 
cedures  to  the  on-the-job  training  process;  and  evaluate  the  AOTS 
system  itself,  providing  trainee  and  system  status  reports  to 
different  levels  of  Air  Force  management. 

1.2.3  AOTS  Computer  Support  Subsystem.  The  Computer  Support 
Subsystem  provides  the  computer  based  tools  and  capabilities  to 
meet  the  objectives  of  the  other  subsystems.  This  subsystem 
provides  the  hardware  and  the  software  products  such  as  editors, 
data  bases,  computational  models,  system  utilities,  and  other 
computer  support,  as  required. 

1.2.4  Key  .Functions.  AOTS  will  include  functional  capabilities 
to  support  the  ongoing  OJT  for  Air  Force  personnel.  The  key 
functions  are: 

a.  Management  of  training  requirements  and  OJT  records 

b.  Development  and  delivery  of  evaluation  objectives  and 
instruments 


3 


c.  Evaluation  of  airmen's  knowledge  and  performance 

d.  Generation  of  reports  documenting  the  AOTS  system  and 
trainee  evaluation  results. 

1.2.5  Interfaces.  As  defined  in  the  AOTS  System  Specification, 
the  AOTS  includes  both  external  and  internal  interfaces.  The  Air 
Force  has  numerous  centers,  groups,  offices,  and  data  bases  that 
maintain  data  vital  to  AOTS  operation.  These  external  offices 
and  data  bases  are  interfaced  with  the  prototype  AOTS  through  off 
line  data  gathering  and  subsequent  on  line  input.  After  the  sys¬ 
tem  is  internally  integrated,  the  AOTS  subsystems  interface  in¬ 
teractively  to  accomplish  system  requirements.  However,  the  AOTS 
Test  and  Evaluation  activities  do  not  impose  any  special  require¬ 
ments  on  these  interfaces. 

1.2.6  Unique  System  Characteristics.  Not  applicable. 

1.3  Key  Technical  Performance  Characteristics. 

The  performance  characteristics  to  be  tested  are  listed  in 
the  Test  Objectives  for  the  various  Test  Plans  in  Appendix  A. 
These  characteristics  derive  from  the  Prime  Item  Development 
Specifications  for  each  subsystem.  Data  to  establish  the 
reliability  and  maintainability  of  the  prototype  system  will  be 
collected  in  accordance  with  the  AOTS  Reliability  Program  Plan 
and  Maintainability  Program  Plan. 

1.3.1  Management  Subsystem. 

a.  Training  requirements  management 

b.  Airman  training  management 
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1.3.2  Evaluation  Subsystem. 


a.  Evaluation  instrumentation  management 

b.  Performance  evaluation 

c.  Training  quality  control 

d.  System  evaluation 

1.3.3  Personnel  and  Support  Subsystem. 

a .  Personnel  support 

b.  Logistics  support 

c.  Maintainability,  reliability,  and  human  factors 

1. 4  Formative  and  Summative  Test  and  Evaluation* 

Formative  evaluation  activities  are  conducted  internally  by 
the  developers  to  determine  the  degree  of  attainment  of  specific 
program  design  goals  and  to  pinpoint  parts  of  the  goals  not  yet 
achieved.  Summative  evaluation,  on  rhe  other  hand,  is  directed 
to  a  more  general  assessment  of  the  degree  to  which  broader 
program  outcomes  have  been  obtained  over  the  entire  program.  The 
distinctions  between  the  two  types  of  evaluation  can  be  explained 
in  terms  of  timing,  level  of  generalization,  and  the  use  of  the 
information  learned.  Formative  evaluation  occurs  during  the 
"formative"  or  development  stage  of  a  program.  Very  specific  in¬ 
formation  about  individual  program  components  is  gathered  to 


*  Formative  Evaluation  and  Summative  Evaluation  are  not 

standard  terminology  for  Air  Force  testing,  but  are  standard 
in  the  literature  on  evaluation  of  training  and  training 
systems. 


develop  the  kind  of  evidence  that  will  be  most  useful  to  the 
developers  in  improving  the  program.  The  term  summative  evalua¬ 
tion  is  used  to  indicate  the  type  of  evaluation  used  after 
development  for  purposes  of  making  judgments  about  the  new 
program's  worth.  Certification  of  outcomes  achieved  or  research 
on  the  effectiveness  of  the  program  (when  compared  to  another)  is 
the  intent  of  the  data  collection  and  analysis  activities. 

The  Formative  Test  and  Evaluation  (T&E)  of  AOTS  during  Phase 
II  development  will  be  primarily  concerned  with  verifying  attain¬ 
ment  of  technical  performance  specifications  and  objectives. 
This  includes  testing  and  evaluating  subcomponents,  components, 
interfaces,  and  subsystems  of  AOTS.  Formative  evaluation  results 
can  then  be  used  to  revise  or  modify  the  system  elements  so  they 
perform  at  or  above  acceptable  levels,  and  so  that  the  integrated 
system  will  be  functionally  complete. 

During  Phase  III,  the  implementation  of  the  system,  summa¬ 
tive  T&E  will  be  accomplished.  The  results  of  the  summative 
evaluation  will  be  used  to  evaluate  the  AOTS,  based  on  program 
outcomes  achieved. 

1.5  Critical  Issues 

Following  are  four  critical  issues  of  overriding  concern  in 
assessing  the  success  of  AOTS.  More  global  in  relation  to  the 
functional  integrity  of  the  system,  they  relate  to  a  system's 
overall  capability  and  must  be  addressed  in  an  evaluation  of  the 
system  as  a  whole.  Furthermore,  they  are  of  primary  importance 
to  the  decision  authority  for  AOTS. 

1.5.1  Critical  Issu^  No.  1;  Compliance.  Does  the  prototype 
AOTS  meet  the  design  and  functional  requirements  of  the  System, 
Subsystem,  and  Component  specifications? 
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1.5.2  Critical  Issue  No.  2:  Performance.  How  does  performance 
of  the  prototype  AOTS  compare  with  the  conventional  OJT  system, 
and  does  AOTS  meet  the  system  performance  standards? 

1.5.3  Critical  Issue  No.  3:  Suitability.  Does  the  AOTS 
overcome  currently  defined  deficiencies  in  the  Air  Force's  OJT 
system,  and  can  AOTS  be  used  for  OJT  throughout  the  Air  Force? 

1.5.4  Critical  Issue  No.  4:  Acceptance.  Is  AOTS  accepted  by 

the  various  system  users  as  user  friendly  and  easy  to  use? 


2.0 


MTP  OVERVIFW 


2 . 1  Approach 

This  Master  Test  Plan  (MTP)  describes  the  overall  strategy 
that  will  be  followed  during  T&E  of  the  prototype  AOTS.  From  an 
evaluative  standpoint ,  the  MTP  is  synonymous  with  program  evalua¬ 
tion.  Furthermore,  it  describes  an  "internal  evaluation"  which 
will  be  planned  and  conducted  by  the  contractor,  in  conjunction 
with  the  Air  Force,  rather  than  an  evaluator  external  to  the 
project. 

An  effective  evaluation  program  is  typically  designed 'as  a 
continual  process  that  remains  in  effect  for  the  program's  dura¬ 
tion  (see  Figure  1)  .  Ongoing  feedback  loops  support  continual 
revision  and  improvement  of  the  program.  Data  are  collected  on  a 
regular  basis,  testing  and  evaluation  take  place,  and  feedback  to 
the  program  occurs.  The  program  is  then  modified  and  the  cycle 
repeats.  The  MTP  must,  therefore,  be  dynamic  rather  than  static 
if  it  is  to  be  effective.  Note  that  the  feedback  loop  from  sum- 
mative  evaluation  is  weaker  than  that  from  formative  evaluation, 
reflecting  the  different  emphasis  in  these  two  types  of  evalua¬ 
tion. 


DESIGN 


TEST  PLANS 


DEVELOPMENT 
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FIGURE  1.  EVALUATION  PROCESS 
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2 . 2  Formative  and  Summative  Evaluation 

A  major  function  of  the  formative  stage  of  evaluation  ac¬ 
tivities  is  to  enable  the  developers  to  examine,  review,  and 
rethink  assumptions  and  designs  that  underlie  the  newly  developed 
program  components.  This  is  to  provide  input  to  decisions  in¬ 
volving  alternative  courses  of  actions  and  settle  controversies 
about  more  effective  ways  to  install  the  program.  If  the  purpose 
of  the  evaluation  is  to  advise  the  program  planners  and 
developers  in  the  actual  development  of  the  program  itself,  a 
formative  evaluation  is  in  order.  Evaluation  activities  are 
designed  to  gauge  potential  problems,  identify  areas  where  the 
program  needs  improvement,  describe  and  monitor  program  ac¬ 
tivities,  and  periodically  test  progress  or  assess  attitudes  of 
the  users. 

The  underlying  purpose  of  the  AOTS  summative  evaluation  is 
to  produce  statements  summarizing  the  program's  accomplishments, 
and  a  summary  statement  about  the  effectiveness  of  the  program, 
in  a  report  to  the  Air  Force.  The  following  information  will  be 
reported: 

o  description  of  the  program 

o  summary  statements  concerning  program  achievement  of 
announced  goals 

o  descriptions  of  any  unanticipated  outcomes 

o  as  possible,  comparisons  with  alternative,  programs 

Since  this  report  could  affect  important  decisions  about  the 
program's  future,  the  evaluations  must  be  able  to  support  the 
findings  and  conclusions.  Although  few  field  evaluations  of 
program  outcomes  can  or  need  to  live  up  to  the  rigid  standards-  of 
true  scientific  experiments,  a  critical  characteristic  of"  any 
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evaluation  study  is  that  it  provides  the  best  information  that 
could  be  gathered  under  the  circumstances,  and  that  this  informa¬ 
tion  meets  the  credibility  requirements  of  its  review  audience. 
The  goal  of  the  evaluator  during  summative  evaluations  is  to 
provide  findings  about  a  program's  effects  that  can  be  general¬ 
ized  to  contexts  other  than  the  one  currently  under  study.  For 
this  reason,  use  of  a  research  design  is  typically  associated 
with  summative  evaluations. 

Formative  evaluations  are  appropriate  for  testing  and  im¬ 
proving  a  prototype  system;  summative  evaluation  is  appropriate 
in  determining  the  worth  of  the  system  or  when  alternate  systems 
are  being  compared  following  the  debugging  and  program  refinement 
period.  It  is  not  unusual,  due  to  time  and  money  constraints, 
for  both  types  of  evaluation  to  occur  very  close  in  time  to  the 
prototype  implementation.  The  problem  arising  in  this  situation 
is  the  possibility  that  projections  about  program  outcomes  and 
effects  based  on  early  implementation  data  are  premature  and  sub¬ 
ject  to  high  error.  However,  program  sponsors  and  users  need 
evidence  on  which  to  base  their  decisions.  It  is  essential  that 
evaluation  audiences  not  attempt  to  use  formative  data  f  »r  summa¬ 
tive  evaluation  purposes.  Summative  evaluation  of  a  program  is 
appropriate  only  in  an  operational  mode. 

Formative  and  summative  evaluation  of  AOTS  will  be  as  shown 
in  Figure  2.  Part  1  of  the  MTP,  as  implemented  during  Phase  II, 
constitutes  formative  evaluation.  Summative  evaluation  (System 
Level  Test  and  Evaluation,  SLT&E)  of  the  prototype  AOTS  is  ac¬ 
complished  in  Phase  III,  as  Part  2  of  the  MTP  is  implemented. 
The  results  of  the  SLT&E,  submitted  in  a  final  report  late  in 
Phase  III,  will  play  a  significant  role  in  establishing  whether 
or  not  the  prototype  AOTS  adequately  addresses  the  critical 
issues  identified  for  AOTS.  Note  that  the  time  distinction  will 
not  be  quite  this  clear  -  some  aspects  of  summative  evaluation. 


especially  baseline  data  collection,  will  have  to  occur  during 
Phase  II. 
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FIGURE  2.  FORMATIVE  AND  SUMMATIVE  EVALUATION  OF  AOTS 


A  first  step  in  the  formation  of  general  plans  for  the  test¬ 
ing  and  evaluation  program  for  AOTS  was  the  identification  of 
each  element  of  the  AOTS  program.  These  elements  are  initially 
defined  at  the  subcomponent  level.  The  subcomponents  are,  in 
curn,  the  elements  of  the  components;  the  components  are  the  ele¬ 
ments  of  the  subsystems;  and  the  subsystems  are  the  elements  of 
the  system.  Because  each  subcomponent  is  an  element,  there  will 
be  as  many  elements  at  the  component  level  as  there  are  subcom¬ 
ponents  and  as  many  elements  at  the  subsystem  level  as  there  are 
components,  and  so  on.  It  is  not  feasible  to  work  with  the  con¬ 
cept  of  an  element  existing  at  all  these  levels.  An  element 
defined  and  tested  at  the  subcomponent  level,  for  example,  will 
"disappear"  at  the  component  level. 

There  must  be  a  clear  hierarchical  structure  in  order  to  lead 
naturally  to  an  evaluation  at  the  integrated  system  level.  This 
structure  will  help  to  keep  the  evaluation  plan  efficient  and 
well  organized.  Test  objectives  that  are  considered  critical  at 
tiie  subcomponent  level  may  seem  relatively  insignificant  when 
compared  to  the  critical  objectives  identified  at  the  subsystem 
or  system  level.  Once  an  element  is  identified,  requirements 
against  it  (the  conditions  it  must  meet)  must  be  specified.  The 
element’s  function  in  the  system  becomes  the  criterion  by  which 


11 


the  performance  of  the  element  may  be  demonstrated  or  verified 
(i.e.  what  are  the  indications  of  the  element  functioning  as 
planned?) .  These  steps  are  strongly  linked  to  the  test  objec¬ 
tives  associated  with  each  element.  Success  criteria,  or  levels 
of  successful  performance,  will  be  the  standard  used  to  verify 
that  the  element  has  "passed"  the  test. 

2 . 3  MTP  Test  Phases 

Part  1  of  the  MTP  defines  the  focus  and  order  of  the  subcom¬ 
ponent,  component,  and  subsystem  levels  of  evaluation  of  the 
AOTS.  It  is  the  basis  of  the  formative  evaluation  of  the  AOTS. 
The  ongoing  T&E  of  the  basic  elements  of  the  system,  as  the  sub¬ 
systems  are  being  developed  during  Phase  II,  will  be  both 
qualitative  and  quantitative  in  nature.  Assessment  of  an 
element's  satisfactory  functioning  may  rely  on  subjective  data, 
objective  data,  or  both.  Indications  of  less  than  satisfactory 
performance  will  be  used  to  focus  revision  efforts  on  the  ele¬ 
ments  of  concern.  Part  1  of  the  MTP  principally  addresses  Com¬ 
pliance  -  do  the  elements  of  AOTS  meet  specifications  (Critical 
Issue  1,  as  described  in  Section  1.5.1). 

Questions  to  be  answered  by  AOTS  T&E  are  summarized  in 
Figure  3.  Basic  evaluation  questions  are  identified  at  the  sub¬ 
component,  component,  and  subsystem  levels  of  the  AOTS. 


LEVEL 

OUESTION 

CRITICAL  ISSUE 

Subcomponents 

Do  manual  and  computer  based 

plans,  procedures,  and  tools 
satisfy  the  design  requirements 
for  AOTS ?  Are  software  products 

supported  by  the  hardware? 

Compliance 

Are  they  user  friendly? 

Acceptance 

Are  they  effective,  etc.  (i.e., 
how  well  do  they  work)? 

Performance 

Suitability 

Components 

Do  the  components  meet  the 
specification  requirements? 

Compliance 

Are  they  user  friendly? 

Acceptance 

Do  the  components  provide 

valid  and  reliable  methodolo¬ 
gies  and  mechanisms  for 
accomplishing  the  intended 

functions? 

Performance 

Suitability 

Subsystems 

Do  the  subsystems  provide  the 
required  functions? 

Compliance 

Are  they  user  friendly? 

Acceptance 

Do  the  subsystems  provide 
the  required  capabilities? 

Performance 

suitability 

FIGURE  3 .  QUESTIONS  AND  LEVELS  OF  EVALUATION 
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During  Part  1  of  the  MTP,  alpha  testing,  beta  testing,  and 
readiness  testing  of  the  AOTS  subcomponents,  components,  and  sub¬ 
systems  will  be  accomplished  (see  Figure  4).  Alpha,  beta,  and 
readiness  testing  are  defined  as  follows: 

o  Alpha  Testing.  Alpha  testing  is  accomplished  by  the 
contractor's  software  development  group  to  determine  that  the 
software  does  function  properly  on  the  AOTS  hardware,  and  that 
together  the  hardware  and  software  provide  the  functions  required 
by  the  AOTS  specifications.  Alpha  testing  includes  the  Unit 
Tests,  Integration  Tests,  and  System  Level  Tests  which  are 
described  in  Appendix  D  (Software  Test  Plan) .  A  software  product 
will  complete  alpha  testing  before  being  released  from  the 
contractor's  software  development  group. 

o  Beta  Testing.  Beta  testing  includes  software  testing 
which  begins  after  alpha  testing  of  a  software  product  is  com¬ 
pleted,  and  testing  of  off  line  products  (processes  and 
procedures) .  The  products  are  inspected,  and  contractor  and  AF 
personnel  (during  Phase  II,  principally  instructional  design 
personnel)  use  and  evaluate  the  products.  Beta  testing  is  in  ac¬ 
cordance  with  Appendix  A  (Part  1  Test  Plans) .  A  product  is  con¬ 
sidered  to  be  "in  production"  after  beta  testing  is  satisfac¬ 
torily  completed. 

o  Readiness  Testing.  Testing  near  the  end  of  Phase  II  to 
demonstrate  to  the  Air  Force  that  products  are  ready  for  use  in 
Phase  III.  Readiness  testing  is  the  responsibility  of  Ball  Sys¬ 
tems  Engineering  Division. 
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FIGURE  4.  FLOW  OF  TESTING  DURING  PHASE  II 
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As  shown  in  Figure  4,  Phase  II  testing  begins  with  the  DAC 
Software  Development  Group.  Software  programs  proceed  through 
the  three  levels  of  alpha  testing  (Unit  Testing,  Integration 
Testing,  and  System  Level  Testing)  in  accordance  with  the 
Software  Test  Plan.  Test  results  are  filed  in  the  software  Unit 
Development  Folders  (UDFs) ,  and  test  reports  in  the  Test  Report 
File  which  DAC  will  maintain  at  its  location  on  Bergstrom  AFB. 
The  UDFs  and  Test  Report  File  will  be  available  for  AF  inspection 
at  any  time.  If  a  problem  is  detected  at  any  level  of  alpha 
testing,  the  software  is  reworked  as  necessary  and  alpha  testing 
begins  again.  Software  which  successfully  completes  one  level  of 
alpha  testing  advances  to  the  next  level  or,  if  alpha  testing  is 
completed,  is  released  for  beta  testing. 

Beta  testing  involves  the  DAC  development  staff,  DAC  and  AF 
users  of  AOTS  (principally  instructional  design  and  Instructional 
Systems  Team  [1ST]  personnel  during  Phase  II)  ,  and  DAC  and  AF  ex¬ 
perts  serving  as  evaluators  and  inspectors.  As  testing  proceeds 
in  accordance  with  the  Test  Plans  (Appendix  A) ,  test  reports  and 
problem  reports  (see  Appendix  B)  are  entered  into  DAC's  Test 
Report  File  at  Bergstrom  AFB.  Documentation  on  unsuccessful 
tests  is  referred  to  the  AF/DAC  Test  Working  Group  (TWG)  for  con¬ 
sideration  and  recommendations.  The  TWG's  recommendations  enter 
into  the  Configuration  Management  process  for  classification  as 
Class  l  or  Class  2  change  (as  defined  in  the  AOTS  Configuration 
Management  Plan)  and  for  approval  or  disapproval.  Approved  Class 
1  changes  are  forwarded  as  Contract  Change  Proposals  to  the  AF 
Program  Contracting  Officer  (PCO) .  If  approved  by  the  PCO,  a 
Contract  Change  Notice  (CCN)  will  result.  If  disapproved  by  the 
Air  Force/DAC  Configuration  Control  Board  (CCB)  or  the  PCO,  the 
documentation  is  archived  in  the  Test  Report  File.  Approved 
changes  which  do  not  require  software  change  are  referred  to 
DAC's  Development  Group  for  rework  and  reentry  into  beta  testing. 
Approved  changes  which  do  require  software  rework  go  to  the  DAC 
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Software  Development  Group,  and  then  into  new  alpha  testing. 
After  a  product  completes  a  rou’  !  of  beta  testing,  it  either  en¬ 
ters  into  the  next  higher  leve.  of  beta  testing  or  continues  in 
use  in  Phase  II  awaiting  Readiness  Testing. 

Readiness  Testing  occurs  during  the  last  two  months  of  Phase 
II.  Whereas  alpha  and  beta  testing  are  conducted  and  witnessed 
principally  by  DAC  (although  AF  observation  and  participation  is 
welcome) ,  readiness  testing  will  be  observed  by  AF  repre¬ 
sentatives  (to  be  designated  by  the  AFHRL  Program  Manager)  . 
Readiness  testing  is  intended  to  demonstrate  to  the  AF  that  the 
AOTS  products  are  ready  for  use  in  Phase  III  System  Level  Test 
and  Evaluation. 


Part  2  of  the  MTP,  developed  and  revised  during  Phase  II  and 
implemented  during  Phase  III,  defines  the  focus  and  order  of  the 
overall  integrated  system  level  evaluation  of  the  AOTS;  outlines 
the  procedural  guides  for  the  summative  evaluation  stage;  and 
provides  procedures  and  schedules  for  data  collection,  data 
analysis,  evaluation,  and  reporting  so  that  overall  performance 
of  the  integrated  system  can  be  assessed.  Execution  of  Part  2  of 
the  MTP  will  examine  how  well  the  AOTS  prototype  accomplishes  its 
design  goals  and  what  benefits  accrue  from  its  implementation. 
Part  2  makes  use  of  pretest/post-test  comparisons  and  non¬ 
equivalent  control  groups  in  order  to  provide  the  required  com¬ 
parisons  . 


As  indicated  in  Figure  1,  the  MTP  may  be  revised  and  updated 
as  DAC  and  the  AF  gain  experience  with  the  evaluation  of  the  ele¬ 
ments  of  the  system.  Some  of  the  Lest  objectives  identified  in 
the  MTP,  as  well  as  some  of  the  indicators  of  the  elements'  per¬ 
formance,  may  be  found  to  be  impractical  or  unnecessary.  On  the 
other  hand,  some  unanticipated  critical  issues  may  emerge,  or 
better  indicators  may  be  found. 
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2 . 4  The  Criteria  Acquisition  Model 

A  classic,  widely  used  model  from  the  program  evaluation 
literature  provides  a  framework  for  evaluation  of  the  AOTS 
program.  The  Criteria  Acquisition  Model*  (CAM)  supports  sys¬ 
tematic,  comprehensive  evaluation  of  a  large  scale  system.  The 
central  notion  of  the  model  is  that  there  are  distinct  evaluation 
stages  in  the  life  of  a  development  product  and  it  is  possible  to 
specify  criteria  that  should  be  met  before  the  product  advances 
to  the  next  stage.  There  are  three  basic  facets  of  the  CAM: 


1.  the  formal  stages  of  the  program  are  specified  and 
described 

2.  che  domains  of  the  specific  criteria  that  will  be  used 
as  the  basis  for  the  evaluation  are  established 

3.  the  various  audiences  for  the  evaluation  are  specified. 

The  CAM  is  usually  depicted  as  a  three  dimensional  matrix 
with  one  facet  along  each  dimension.  Each  facet  is  broken  down 
into  several  levels  or  categories.  The  model  is  quite  general 
and  very  flexible.  For  example,  depending  on  the  stage  of  the 
program  during  which  the  evaluation  takes  place,  different  levels 
of  emphasis  may  be  placed  on  the  criteria  domains  or  the 
audiences  because  certain  criteria  or  audiences  are  more  impor¬ 
tant  than  others  at  certain  stages.  Each  stace  in  the 
development/evaluation  sequence  represents  a  relatively  easily 
identified  milestone  in  the  life  of  a  project.  Moreover,  the 
entry  of  a  test  element  into  a  given  stage  can  be  viewed  as  a 
decision  point.  If  a  test  element  lacks  certain  characteristics 


*  Wright,  W.J. ,  &  Hess,  R.J.  (1974) ,  A  criteria  acquisitic 
model  for  educational  evaluation.  In  G.D.  Borich  (Ed.), 
Evaluating  Educational  Programs  and  Products.  Englewood 
Cliffs,  NJ:  Educational  Technology  Publications. 
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or  has  failed  to  meet  certain  criteria  required  for  entry  into 
the  next  stage,  revision  may  be  indicated.  Evaluative  feedback 
will  be  provided  at  these  milestones  for  Air  Force  review.  It  is 
the  stages  of  development  and  evaluation  that  constitute  the 
major  milestones  or  decision  points  in  the  life  of  a  product. 
According  to  the  extent  to  which  a  product  meets  the  criteria  es¬ 
tablished  for  it  at  a  given  stage,  it  will  be  advanced  or 
referenced  to  the  contractor  for  change.  Deciding  how  much 
weight  is  to  be  applied  to  any  criterion  is,  of  course,  a  product 
specific  decision. 

The  three-dimensional  CAM  model  tailored  for  use  in  the  AOTS 
program  evaluation  is  depicted  in  Figure  5.  The  stages  and 
audiences  should  be  self  explanatory,  and  the  criteria  domains, 
listed  below,  are  the  critical  issues  identified  in  Section  1.5. 

2.4.1  Compliance.  Does  the  prototype  AOTS  meet  the  design  and 
functional  requirements  of  the  System,  Subsystem,  and  Component 
Specifications? 

2.4.2  Performance.  How  does  performance  of  the  prototype  ACTS 
compare  with  the  conventional  OJT  system,  and  does  AOTS  meet  the 
system  performance  standards? 

2.4.3  Suitability .  Does  the  AOTS  overcome  currently  defined 
deficiencies  in  the  Air  Force's  OJT  system,  and  can  AOTS  be  used 
for  OJT  throughout  the  Air  Force? 

2.4.4  Acceptance.  Is  AOTS  perceived  by  the  various  system  users 
as  easy  to  use  and  user  friendly? 


2 . 5  Definitions  and  Descriptions  of  the  Basic  Assessment  Process. 
Processes  identified  in  the  requirements  specifications  for  the 
MTP  for  assessing  the  performance  of  the  system  elements  are: 
testing,  measurement,  analysis,  demonstration,  evaluation,  in¬ 
spection,  and  reporting.  Interpreting  how  these  assessment 
process  terms  can  be  implemented  in  practice,  the  seven  processes 
can  be  collapsed  into  the  four  basic  assessment  processes  that 
are  being  required  in  the  MTP.  The  seven  processes  are  as  fol¬ 
lows. 


Inspection.  This  term  refers  to  the  examination  of  a  system 
or  an  element  of  the  system  by  visual  means,  and  implies  the 
physical  presence  of  an  observer  (i.e.,  the  inspector).  Inspec¬ 
tion  also  implies  standards  against  which  the  element  or  system 
is  to  be  compared. 

Analysis.  This  process  refers  to  breaking  down  a  system 
into  its  component  parts,  and  usually  requires  that  an  examina¬ 
tion  be  conducted  of  the  system,  its  component  parts,  and  the  in¬ 
terrelations  among  the  parts  in  forming  the  whole.  Analysis  is 
often  necessary  in  trouble  shooting  a  system  to  identify  the  com¬ 
ponent  causing  problems  so  that  the  faulty  component  can  be  cor¬ 
rected.  Data  analysis  is  a  specific  technique  involving  descrip¬ 
tive  and/or  inferential  statistics  to  describe,  summarize,  pre¬ 
dict,  or  evaluate  some  system  or  measures  of  system  performance. 
Thus,  analysis  often  connotes  analysis,  gathering,  and  inter¬ 
pretation  of  data. 

Test.;.  A  test  is  basically  a  procedure  conducted  to  deter¬ 
mine  whether  some  element  works.  That  is,  does  the  element  func¬ 
tion  as  it  was  designed  to  function?  The  result  of  the  test  is 
usually  a  dichotomous  decision  (e.g.,  yes  or  no),  but  the  test 
may  also  produce  quantitative  measures.  In  all  cases,  the  test 
determines  whether  the  element  conforms  to  the  requirements  •  or 
performance  characteristics  specified  for  it. 


Evaluation.  This  term  refers  to  the  process  by  which  value 
judgments  or  decisions  are  made  through  the  use  of  a  variety  of 
evidence  including  tests,  measures,  demonstrations,  inspections, 
etc.  Evaluation  requires  both  qualitative  and  quantitative  in¬ 
terpretation  of  the  cumulative  weight  of  evidence  to  determine 
the  worth  or  value  of  a  system.  Evaluation  usually  takes  place 
in  a  less  controlled  environment  than  tests  or  measurements,  such 
as  field  evaluations.  Evaluation  also  implies  some  standards  of 
performance  to  be  met,  a  set  of  criteria  to  decide  if  the  stand¬ 
ard  has  been  exceeded,  and  often  includes  comparisons  of  the  sys¬ 
tem  being  evaluated  with  alternate  systems.  Filially,  evaluation 
typically  involves  the  descriptive  or  inferential  statistical 
analysis  of  individual  or  composite  measures  of  system  perfor¬ 
mance. 

Measurement .  Measurement  involves  the  assignment  of  numbers 
(quantitative  values  that  indicate  magnitude,  amount,  degree, 
quantity,  etc.)  to  objects,  events,  systems,  or  persons  according 
to  some  rule.  In  the  measurement  of  physical  characteristics 
such  as  height  or  weight,  the  quantification  rules  have  been 
agreed  upon  and  standardized  so  that  everyone  knows  the  proce¬ 
dures  to  be  followed.  In  other  contexts,  however,  the  rule  will 
usually  have  to  be  specified  in  greater  detail.  It  is  important 
to  remember  that  what  is  measured  are  the  attributes,  traits,  or 
characteristics  of  the  objects,  events,  or  systems,  not  the  ob¬ 
jects,  events,  or  systems  themselves.  For  example,  the  ef¬ 
ficiency,  the  user  friendliness,  the  costs,  the  reliability,  and 
the  frequency  of  use  of  a  system  are  all  attributes  that  might  be 
measured. 

Demonstration .  This  term  refers  to  the  process  of  showing 
some  aspect,  attribute,  or  property  of  a  system.  The  merits, 
capabilities,  capacities,  and/or  performance  levels  of  a  system 
are  displayed  or  illustrated  to  prove  its  effectiveness.  "This 
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overt  display  may  involve  the  use  of  simulations,  explanations, 
or  examples  to  make  the  demonstration  convincing. 

Reporting .  Reporting  is  the  effective  communication  of 
evaluation  results  to  the  audiences  requiring  the  information. 
Reports  may  be  verbal  or  written. 

Given  these  definitions,  it  follows  that  the  process  of 
measurement  is  always  a  part  of  testing  and  analysis;  reporting 
(communicating  evaluation  results)  is  a  part  of  the  general 
evaluation  process;  and  demonstration  may  be  viewed  as  a  proce¬ 
dural  step  within  evaluation.  The  four  remaining  processes',  in¬ 
voked  by  the  Part  1  Test  Procedures,  are  Inspection,  Analysis, 
Test,  and  Evaluation.  The  operational  definitions  and  repre¬ 
sentative  tools  and  analyses  associated  with  each  are  shown  in 
Figure  6. 

2 • 6  Organization  and  Responsibilities. 

Organizations  involved  in  AOTS  T&E ,  and  their  respon¬ 
sibilities,  are  as  shown  in  Figure  7. 

2.6.1  DAC  Responsibilities.  DAC  is  principally  responsible  for 
designing,  conducting,  and  documenting  the  tests  and  evaluations, 
and  for  taking  necessary  corrective  actions.  Under  the  DAC 
Program  Manager,  the  DAC  Test  Manager  has  overal)  responsibility 
for  all  T&E  activities  described  in  this  MTP.  The  DAC  Software 
Manager  has  principal  responsibility  for  corrective  actions  on 
software  products.  The  DAC  Manager  of  Configuration  Management, 
and  the  joint  AF/DAC  CCB,  will  be  involved  in  any  changes  which 
arise  from  T&E  and  which  impact  a  product  under  configuration 
control . 
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LEVEL  OF 

TEST/ 

EVALUATION 

TYPE  OF  TEST 

CONDITIONS 

OF 

TEST 

- - 

CRITERION  MEASURE 

TOOLS/ANALYSES 

Inspection 

Visual  examination  or  equivalent 

Controlled  or 

uncontrolled 

Established  standards  of 
performance  by  expert 
consensus  or  equivalent. 

Checklist. 

Rating  scale. 

Item  by  item  adherence 
or  composite  score  with 
equal  or  varying  weiqhts 
applied. 

Analysis 

Descriptive  data  gathering,  and 
interpretation  techniques 

Controlled  or 

uncontrolled 

Quantifiable  measure  or 
composite  measures 

about  x  (e.g.  '  opinions, 
freq.  of  occurrence,  etc.) 

Attitude  scale 
Questionnaire 

Pecf.  measures 
-  opinion  or  perfor¬ 
mance  data  collection 
and  descriptive  data 
analysis:  summary  of 
composite  values  .  .  . 
“a  profile". 

T6st 

Specified  inputs,  outputs,  and 
data  analysis  methods 

Controlled 

Quantified  measure  or 
composite  measures  to  a 
specified  probability  level 
(i  e.  confidence  levels, 
significance  against  criteri¬ 
on) 

Procedure  that  deter¬ 
mines  if  x  works,  as  far 
as  confonmng  to  re¬ 
quirements  specified  for 
it.  at  a  statistically 
acceptable  confidence 
level. 

Evaluation 

Specified  inputs,  outputs,  and 
data  analysis  methods: 

o  variation  of  input  and 
determination  of 
differential  outputs  and 
consequences 

o  descriptive  or 

statistical  treatment 
of  individual  or 
composite  data 
measures. 

demonstration:  act  of 
proving/illustrating  worth 
of  x:  providing  evidence 

evaluation:  orocess  bv  which 
value  judgements  are  made 
based  on  a  variety  of 
evidence:  to  ascertain  or  fix 
a  value  or  worth 

Controlled  or 
uncontrolled 
(usually  less 

controlled,  e.g. 
field  evaluations) 

Quantifiable 

o  measure 
o  composite 
measure 
o  index 
o  profile 
o  configuration 

Quantitative  and  qualita¬ 
tive  interpretation  of 
cumulative  weight  ol 
evidence,  and  judgment 
as  to  whether  or  not 
standards  of  perfor¬ 
mance  have  been  met 
or  exceeded:  may  in¬ 
clude  comparisons  with 
alternate  systems, 

judgements 

FIGURE  6. 
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2.6.2  AFHRL  Responsibilities.  AFHRL  participates  in  planning 
and  approves  plans  for  T&E,  observes  testing,  and  reviews  test 
results.  AFHRL* s  Instructional  Systems  Team  and  Concepts  and 
Analysis  Branch,  other  AF  personnel  and  organizations  including 
the  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC) , 
and  Ball  Systems  Engineering  will  be  involved  in  the  T&E  ac¬ 
tivities  as  the  AFHRL  Program  Manager  may  direct. 

2.6.3  Test  Working  Group.  The  purpose  of  the  Test  Working  Group 
(TWG)  is  to  provide  a  forum  for  continuous  interchange  on  test 
related  issues  among  organizations  involved  in  the  test  and 
evaluation  planning  process.  The  TWG,  chaired  by  the  DAC' Test 
Manager,  will  advise  the  DAC  and  AFHRL  Program  Managers  on  all 
significant  test  matters.  The  TWG  will  operate  throughout  the 
period  of  AOTS  Test  and  Evaluation. 

2 *6. 3.1  TWG  Responsibilities.  Responsibilities  of  the  TWG  are 
summarized  below.  These  responsibilities  will  be  revised  or  ex¬ 
panded  as  program  requirements  may  dictate. 

a.  Review  and  comment  on  formative  and  suinmative  test 
objectives,  test  evaluation  criteria  baselines,  T&E 
organizational  responsibilities,  and  test  documentation 
requirements . 

b.  Review  and  comment  on  policy  and  plans  for  major  test 
milestones. 

c.  Review  and  comment  on  test  plans  to  determine  that  test 
objectives  and  requirements  are  addressed. 

d.  Review  and  comment  on  the  Master  Test  Plan  (AFHRL  as  the 
procuring  agency  has  responsibility  and  authority  for 
approval) . 
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e.  Identify  test  support  requirements  to  appropriate 
organizations,  and  recommend  alternate  test  methods  or 
objectives  when  specific  test  resources  are  not 
available. 

f.  Maintain  current  knowledge  of  the  critical  T&E  issues, 
test  objectives,  progress  toward  attainment,  and 
resolution  of  deficient  areas.  In  cases  requiring 
interpretation  of  requirements  and/or  results,  the  TWG 
will  make  recommendations  to  DAC  and  HRL  management  and 
to  the  CCB. 

g.  Review  program  redirection  and  the  effects  on  current 
T&E  costs,  performance,  schedule,  and  resource 
estimates. 

2. 6. 3. 2'  TWG  Membership.  The  TWG  will  consist  of  designated  fo¬ 
cal  points  from  each  participating  organization,  as  follows: 

a.  Chairperson:  DAC  Test  Manager 

b.  Vice  chairperson:  DAC  Program  Manager 

c.  Permanent  Members: 

-  AFHRL  Program  Management  Representative  (1) 

1ST  Representatives  (2) 

DAC  Representatives  (2) 

Ball  Systems  Engineering  Representative  (1) 

d.  Special  Advisors:  From  time  to  time  the  TWG  Chairperson 
may  request  other  individuals  to  assist  with  or 
participate  in  the  group’s  activities.  These  persons 
will  not  be  considered  permanent  TWG  members. 
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2. 6. 3. 3  TWG  Procedures. 


a.  Meetings  of  the  TWG  will  be  held  quarterly,  at  Bergstrom 
AFB  locations  and  at  times  to  be  determined  by  the  Chairperson. 
Insofar  as  possible,  meetings  will  be  scheduled  when  a  majority 
of  TWG  members  will  be  available.  Any  permanent  TWG  member  may 
request  a  meeting  to  discuss,  plan,  and  resolve  T&E  issues. 

b.  Meeting  minutes  will  be  distributed  to  members  of  the 
TWG  and  to  DAC  and  AFHRL  program  management  within  10  days  after 
each  meeting.  Minutes  will  outline  the  meeting  objectives,  the 
major  results,  action  items,  and  recommendations  made  by  the 
group.  The  Chairperson  will  appoint  a  recorder  for  each  session. 

c.  Documents  requiring  TWG  action  will  be  provided  to  the 
members  Sufficiently  in  advance  of  a  meeting  to  allow  adequate 
review  time. 


3.0 


MTP  PART  1:  PHASE  II  TEST  AND  EVALUATION 


3 . 1  Introduction 

As  described  above,  T&E  activities  for  Part  1  of  the  MTP  are 
formative  in  nature.  The  principal  objective  for  Part  1  is  to 
test  each  subcomponent,  component,  and  subsystem  of  AOTS  for  ad¬ 
herence  to  functional  requirements  and  for  acceptability.  The 
performance  characteristics  and  functions  identified  in  system 
and  subsystem  specifications  become  the  testing  objectives  for 
subcomponents,  components,  and  subsystems. 

3 . 2  Formative  Level  Test  Objectives 

Part  1  of  the  MTP,  formative  evaluation  of  AOTS,  is  an  ongo¬ 
ing  qualitative  and  quantitative  assessment  while  subsystems  are 
being  developed.  Each  element  of  the  system  (i.e.,  subcomponent, 
component,  and  subsystem)  is  identified  here.  The  requirements 
for  each  element  to  be  tested  are  stated.  In  some  cases,  the 
requirements  are  qualitative,  e.g.,  "an  element  shall  include  X, 
Y,  and  Z,"  and  in  some  cases  are  quantitative,  e.g.,  "the  element 
shall  perform  its  function  with  98%  accuracy."  Testing  proce¬ 
dures  and  instruments  will  be  unique  for  each  test.  Refer  to  Ap¬ 
pendix  A,  the  individual  test  plans  for  each  element  identified 
in  the  T&E  plans,  for  specific  test  objectives,  evaluation  ques¬ 
tions,  and  methods  of  evaluation.  General  descriptions  of  proce¬ 
dural  guides  to  be  developed  for  test  activities  are  contained  in 
Appendix  B. 

3 • 3  Order  of  Testing. 

Refer  to  Appendix  G  of  this  document  for  the  testing  mile¬ 
stones  and  the  order  in  which  tests  of  the  system  elements  will 
be  performed.  Beta  testing  of  an  element  will  not  begin  until 
the  associated  alpha  testing  of  that  element  has  been  completed. 

Since  AOTS  hardware  will  be  purchased  off  the  shelf, 
hardware  testing  will  be  done  using  hardware  tests  supplied  by 
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the  manufacturer.  There  will,  therefore,  be  no  AOTS  unique 
hardware  tests.  In  all  cases,  specific  hardware  tests  will  be 
conducted  prior  to  initiating  MTP  test  activities  whose  results 
might  be  constrained  by  the  operation  of  that  hardware.  Details 
of  hardware  testing  procedures  are  contained  in  Specifications 
70S647401  and  70S647402.  Software  required  to  support  an  MTP 
beta  test  activity  will  complete  alpha  testing  prior  to  beginning 
the  MTP  beta  test  activities.  Specific  details  of  software  test¬ 
ing  procedures  (alpha  testing)  are  documented  in  the  Software 
Test  Plan,  Appendix  D  of  this  MTP. 

Whenever,  during  MTP  testing,  a  problem  or  deficiency  is 
noted  and  the  cause  is  not  apparent,  both  hardware  and  software 
will  be  considered  as  possible  contributors.  If  problems  with 
hardware  are  found  and  the  solution  is  outside  the  manufacturer's 
stated  capabilities  for  said  hardware,  then  the  Air  Force  will  be 
consulted  for  mutual  resolution.  If  the  problem  is  related  to 
software,  the  programmers  will  review  the  problem,  and  keep  the 
Air  Force  appraised  of  the  proposed  resolution. 
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4.0  MTP  PART  2:  SYSTEM  LEVEL  TEST  AND  EVALUATION 


The  Air  Force  Human  Resources  Laboratory  (AFHRL)  has  been 
tasked  to  design,  develop,  and  test  a  prototype  Advanced  On-the- 
job  Training  System  for  Air  Force  On-the-Job  Training  (OJT) .  The 
prototype  system  is  to  be  demonstrated  and  evaluated  in  an  opera¬ 
tional  environment.  Four  Air  Force  Specialties  (AFSs)  have  been 
selected  to  participate  in  this  effort.  These  are  426X2,  Jet  En¬ 
gine  Maintenance;  431X1,  Aircraft  Maintenance;  732X0,  Personnel; 
and  811XX,  Security  Police.  All  Air  Force  components  (Active, 
Reserve,  and  Guard)  will  participate  in  this  effort.  Bergstrom 
AFB,  TX  has  been  selected  as  the  site  at  which  the  Active  and 
Reserve  Forces  will  participate,  and  Ellington  ANGB,  TX  is  the 
site  selected  for  the  Guard  Forces.  The  workcenters  involved,  by 
AFS  at  each  site,  are  listed  in  Appendix  E. 

The  McDonnell  Douglas  Corporation  was  awarded  a  contract  for 
developing- the  prototype  AOTS  in  August  1985.  Time  for  the  ef¬ 
fort  is  four  years;  August  1985  -  July  1989.  The  effort  is 
divided  into  three  phases:  Phase  I  fbr  preliminary  design  of  the 
prototype  system;  Phase  II  for  detailed  design  and  development; 
and  Phase  III  for  implementation,  demonstration,  and  System  Level 
Test  and  Evaluation  (SLT&E) . 

This  Part  2  of  the  Master  Test  Plan  (MTP)  outlines  the 
requirements  to  be  met  during  the  SLT&E.  Included  are  test  plans 
for  the  critical  issues  of  Compliance,  Performance,  Suitability 
and  Acceptance;  surveys  to  be  completed  by  the  Air  Force  users; 
and  the  SLT&E  time  schedules. 

4 . 1  Background  and  Authority 

4.1.1  System.  The  AOTS  grew  out  of  an  Air  Staff-directed  func¬ 
tional  management  inspection  of  the  Air  Force  OJT  program  in  the 
late  1970s.  The  inspection  found  a  number  of  problems  and_  sum¬ 
marized  the  current  system  as  being  labor-  and  paper-intensive 
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and  having  limited  use  of  automation,  which  is  focused  on  record 
keeping  instead  of  training  and  readiness.  The  Air  Staff 
directed  a  study  of  possible  solutions.  One  was  an  initial  plan 
for  an  automated  unit-level  training  system  that  eventually  be¬ 
came  AOTS . 

4.1.2  Acquisition  Program.  The  Air  Force  Systems  Command  was 
directed  (RPR  80-03,  PMD  2029,  most  recent  update  9  Oct  86)  to 
design,  develop,  and  test  a  prototype  for  a  job-site  training 
system.  The  system  was  to  address  training  concerns  such  as 
responsiveness,  standardizing  instruction  and  evaluation,  train¬ 
ing  proficiency,  and  applying  computer-based  training  delivery 
and  management.  Air  Force  Systems  Command  assigned  the  Air  Force 
Human  Resources  Laboratory  (AFHRL)  Training  Systems  Division  to 
do  the  work.  In  1985,  AFHRL  contracted  with  McDonnell  Douglas 
Corporation's  Douglas  Aircraft  Company  (DAC)  for  a  4-year  program 
to  design,  develop,  test  and  evaluate  the  prototype  AOTS.  Phase 
I,  preliminary  design,  was  completed  in  May  1986;  Phase  II, 
detailed  design  and  development,  will  be  completed  in  July  1988; 
and  Phase  III,  implementation  of  the  prototype,  will  begin  in 
August  1988.  System  Level  Test  and  Evaluation  (SLT&E)  will  begin 
in  August  1988  and  will  be  concluded  in  July  1989. 

Tost — Program...  The  Air  Staff  sponsor  for  the  project,  HQ 
USAF/DPP ,  directed  that  the  AOTS  prototype  be  developed  and 
evaluated  within  an  operational  environment  to  ensure  that  the 
system  is  useful  and  acceptable  to  the  noncommissioned  officer 
(NCO)  force.  CONUS  MAJCOMs  recommended  and  AFHRL  assessed  a  set 
of  bases.  HQ  USAF/DPP  selected  Bergstrom  AFB,  Texas,  using 
criteria  which  included  the  following; 
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(a)  a  representative  flying  mission 

(b)  total  force  units  (active  Air  Force,  Air  Force  Reserve, 
and  Air  National  Guard)  in  close  proximity 

(c)  all  units  of  the  total  force  components  fly  the  same 
weapon  system;  and 

(d)  stable  weapon  system 

The  prototype  will  be  implemented  and  evaluated  in  work  cen¬ 
ters  of  the  following  units: 

Component _ Unit _ Weapon  System 

Active  Air  Force  67  TRW  RF-4C 

Bergstrom  AFB,  TX 

Air  Force  Reserve  924  TFG  F-4D 

Bergstrom  AFB,  TX 

Air  National  Guard  147  FIG  F-4D 

Ellington  ANGB,  TX 

Before  the  start  of  Phase  III,  DAC  will  conduct  alpha  and 
beta  testing.  The  Air  Fojcce  will  conduct  readiness  testing  (MTP 
Part  1)  to  ensure  that  the  AOTS  works  before  it  is  placed  in  the 
operational  workcenters. 

The  AFSs  participating  in  the  AOTS  prototype  were  selected 
for  a  number  of  reasons,  including  (a)  they  are  large  (together 
they  include  approximately  20%  of  the  enlisted  force) ;  (b)  they 

represent  a  wide  range  of  job  types;  and  (c)  the  OJT  for  each  is 
somewhat  different.  Together  they  should  provide  a  realistic 
test  of  the  usefulness  of  the  AOTS.  Also,  the  functional  areas 
including  each  of  these  specialties  have  either  an  existing  or 
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planned  automation  system  (CAMS,  MMICS,  PC-III,  and  SPAS).  This 
offers  an  opportunity  to  integrate  AOTS  and  these  representative 
automation  systems. 

4.1.4  Program  Documentation. 

1.  AFHRL-TP-83-54,  Integrated  Training  System 

for  Air  Force  On-the-job  Training:  Specification 
Development.  March  1984. 

2.  AOTS  Statement  of  Work.  16  April  1984. 

3.  Program  Management  Directive  (PMD)  2029.  9  October 

1986. 

4 .  Memorandum  of  Agreement  (MOA)  Between  AFHRL/HQ  AMD  and 
HQ  TAC,  HQ  ATC,  HQ  Air  University,  ANG,  AFRES,  AFMPC, 
AFOSP,  AF/MPPT,  and  AF/LEYM.  July  1984. 

4 . 2  SLT&E  Description 

4.2.1  Purposes .  Four  major  evaluation  areas  (critical 

issues)  are  to  be  addressed  during  SLT&E:  Compliance,  Perfor¬ 

mance,  Suitability,  and  Acceptance.  The  purposes  of  the  SLT&E 
are  to  show  that  DAC  meets  the  conditions  of  the  contract  and 
specifications  (Compliance) ,  and  to  evaluate  the  AOTS  in  the 
operational  environment  by  comparing  it  to  existing  OJT  and  dis¬ 
covering  any  problems  or  deficiencies  with  AOTS  (Performance,  Ac¬ 
ceptance,  and  Suitability) . 

4.2.2  Approach .  The  elements  of  the  system  that  will  be 

tested  and  evaluated  will  depend  on  the  evaluation  area  being 
addressed.  Also,  the  target  groups  to  participate  in  the  test 
and  evaluation  and  the  methods  for  gathering  data  will  vary 
depending  on  the  element  being  tested  or  evaluated.  Target 
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groups  refer  to:  the  Instructional  Systems  Team  (1ST) ,  both  Air 
Force  and  Douglas  Aircraft  Company  members;  the  supervisors, 
trainers,  trainees  and  evaluators  assigned  to  the  applicable  work 
centers;  and  commanders  and  training  managers.  The  methods  to  be 
used  for  gathering  data  include  interviews,  surveys,  and  collec¬ 
tion  of  statistical  data. 

Interview  and  survey  instruments,  and  procedures  for  ad¬ 
ministration,  are  contained  in  Appendix  J. 

The  test  plans  which  follow  (Appendix  E)  include,  in  matrix 
format  for  each  evaluation  area,  the  following  information: 

1)  the  critical  questions  and  subquestions  to  be  addressed 
for  each  critical  issue 

2)  the- method  or  methods  to  be  used  for  measurement  for  each 
subquestion, 

3)  the  data  to  be  collected  and  the  target  groups  to 
participate 

4)  how  data  will  be  collected 

5)  how  data  will  be  analyzed 

6)  how  the  results  of  analyses  will  be  evaluated 

7)  how  the  results  of  data  analysis  and  evaluation  will  be 
reported,  and 

8)  the  elements  being  tested. 

During  SLT&E ,  DAC  will  cooperate  with  base  OJT  managers  to 
keep  them  aware  of  activities  and  to  include  them  in  the  coor- 
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dination  loop  with  ACTS  activities.  AOTS  will  support  scheduling 
of  FTD  courses  through  the  MAT.  MAT  training  managers  will  be 
asked  to  complete  questionnaires  during  SLT&E  to  respond  to 
critical  issues  of  Acceptance  and  Suitability. 


4 . 3  Program  Schedules  and  Milestones 

The  milestones  associated  with  SLT&E  are  shown  in  Appendix  G 
of  this  MTP.  Appendix  G  also  contains  flowcharts  showing  the  or¬ 
der  in  which  interviews,  surveys,  data  collection,  analysis, 
evaluation,  and  reporting  will  occur,  and  the  timelines  for  each. 

4 . 4  System  Description 

The  AOTS  is  a  computer-based  system  that  will  administer, 
evaluate,  track,  and  manage  OJT.  Refer  to  MTP  Part  1,  Sections 
1.0 — 1.3.3  for  a  more  detailed  description. 

The  test  hardware  for  Phase  III  will  include  Zenith  Z-248 

X 

Personal  Computers;  a  VAX  8600  mainframe  computer  located  at 
Brooks  AFB  in  San  Antonio,  Texas;  a  56KB  high-speed  communica¬ 
tions  line  between  Brooks  AFB  and  Bergstrom  AFB;  printers;  opti¬ 
cal  mark  readers;  digitizing  tablets;  and  other  associated 
equipment  necessary  for  AOTS  operation.  During  SLT&E,  the  system 
will  be  maintained  in  accordance  with  the  AOTS  Maintenance  Plan. 

4 . 5  Scope  and  Limiting  Factors 

4.5.1  Locations.  This  test  will  be  conducted  in  workcenters  at 
Bergstrom  AFB  and  Ellington  ANGB.  The  following  specific  loca¬ 
tions  will  be  used: 

1 .  Bergstrom  AFB 

a.  Jet  Engine  Maintenance  Workcenters 

Active  -  Bldgs  1612  and  4529;  Reserves  -  Bldg  4589 
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b.  Aircraft  Maintenance  Workcenters 

Active  -  Bldgs  1609  and  4529;  Reserves  -  Bldg  4515 

c.  Personnel  Workcenters 

Active  -  Bldg  2202;  Reserves  -  Bldg  4555 

d.  Security  Police  Workcenters 

Active  -  Bldgs  207  and  208;  Reserves  -  Bldg  4204 

e.  AFHRL/OL-AK  Program  Management 
Trailer  T-l 

f.  AFHRL/OL-AK  Instructional  Systems  Team 
Bldg  1808 

g.  DAC  Management  and  Programming 
Bldg  428 

h.  Chiefs  of  Maintenance  Training  Staffs 
Active  -  Bldg  1501;  Reserve  -  Bldg  4592 

2.  Brooks  AFB 

a.  Base  Information  Processing  Center  (VAX  8600) 

Bldg  578 

3.  Ellington  ANGB 

a.  Jet  Engine  Maintenance  Workcenters 
Bldg  1290 

b*  Aircraft  Maintenance  Workcenters 
Bldg  1382 

c.  Personnel  Workcenters 
Bldgs  1057  and  1382 
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d.  Security  Police  Workcenters 
Bldg  1193 

4.5.2  Personnel  Requirements.  Conducting  this  test  will  require 
participation  of  the  following  personnel: 

1.  AFHRL  personnel,  including  1ST  members,  at  Bergstrom  AFB 

2.  Workcenter  personnel  (AOTS  and  control  groups)  at 
Bergstrom  AFB  and  Ellington  ANGB  (Figures  8,  9,  and  10 
further  illustrate  the  participation  of  each  group.) 


3 .  DAC  Employees  at  Bergstrom  AFB 

.  Estimates  of  the  time  required  from  AF  personnel  for  SLT&E 
activities  are  included  in  the  test  plans  in  Appendix  E. 


4.5.3  Limiting  Factors.  Planning  for  SLT&E  of  AOTS  is  in¬ 
fluenced  strongly  by  a  variety  of  practical  constraints  and 
limitations.  The  major  factor  is  that  the  nature  of  the  im¬ 
plementation  of  AOTS,  as  a  small  scale  prototype,  dictates  that  a 
"field  research"  approach  be  taken.  The  evaluation  will  take 
place  in  the  operational  Air  Force  environment,  precluding  the 
possibility  of  a  more  controlled  laboratory  or  experimental  ap¬ 
proach  (along  with  the  advantages  and  benefits  of  such  evaluation 
methods) .  The  objective  is  to  accomplish  the  best  possible 
evaluation  of  AOTS  within  the  constraints  imposed  by  the  opera¬ 
tional  environment.  The  Master  Test  Plan  therefore  adopts  quasi- 
experimental  designs  for  use  in  SLT&E  (cf.  Campbell  &  Stanley, 
Experimental  and  Ouasi-Experimental  Designs  for  Research.  Rand 
McNally,  1966) .  As  indicated  in  the  Test  Plans  (Appendix  E)  , 
nonequivalent  control  group  designs  will  be  used  where  ap¬ 
propriate  control  groups  exist  at  Bergstrom  AFB.  Where  suitable 
control  groups  are  lacking,  time  series  designs  will  be  used-  to 
compare  the  pre-AOTS  baseline  with  AOTS. 
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Any  consideration  of  the  features  and  design  of  the  MTP 
needs  to  bear  in  mind  the  restrictive  conditions  under  which 
testing  will  have  to  be  conducted.  The  major  limitations  on  the 
MTP  are  summarized  below,  in  approximate  order  of  importance  or 
impact  on  the  SLT&E . 

Lack  of  Randomization  -  The  AOTS  prototype  will  be  imple¬ 
mented  in  only  a  few  workcenters,  for  four  Air  Force  Specialties, 
at  two  bases.  Bergstrom  AFB  and  Ellington  ANGB  were  not  selected 
randomly,  nor  were  the  four  AFSs  or  the  wci  kcenters  selected 
randomly  from  among  all  AFSs  or  workcenters.  Furthermore,  sub¬ 
jects  to  be  studied  in  Personnel,  Security  Police,  Jet  Engine 
Maintenance,  and  Aircraft  Maintenance  workcenters  comprise  con¬ 
venience  samples,  self-selected  to  some  degree,  rather  than  ran¬ 
dom  Samples .  It  will  not  be  possible  to  assign  airmen  to  treat¬ 
ment  conditions  randomly.  Instead,  intact  workcenters  will  be 
measured  during  a  baseline  periol  prior  to  the  installation  of 
AOTS;  again  during  the  implementation  of  the  AOTS  prototype;  and 
in  control  groups.  The  lack  of  random  selection  and  assignment 
of  subjects  eliminates  the  possibility  of  conducting  any  type  of 
true  experimental  design. 

Control _ Groups  -  There  are  practical  considerations  which 

limit  the  availability  and  use,  at  Bergstrom  AFB,  of  control 
groups  of  non-AOTS  participants.  In  the  case  of  Jet  Engine  In¬ 
termediate  Maintenance,  the  only  such  workcenter  at  Bergstrom  AFB 
will  be  under  AOTS.  In  other  cases,  even  though  there  may  be 
similar  workcenters  at  Bergstrom,  duties  differ  considerably. 
Also,  personnel  may  be  transferred  from  one  workcenter  to  another 
during  the  course  of  AOTS,  thus  confounding  the  groups. 

Practical  considerations  and  costs  limit  the  usefulness  of 
control  groups  from  AFBs  other  than  Bergstrom.  Such  groups  need 
to  come  from  an  AFB  similar  in  size  and  functions  -  same  type, 
number,  and  configuration  of  weapon  systems,  same  positions  and 
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tasks,  similar  environment  and  geographic  location,  etc.  The 
near  certainty  of  marked  differences  in  these  respects  from  AOTS 
workcenters  at  Bergstrom  AFB  argues  against  this  approach. 

Non-independence  of  Groups  -  Because  all  of  the  active  and 
reserve  personnel  participating  in  AOTS  are  located  at  Bergstrom 
AFB  (and  all  of  the  Air  National  Guard  personnel  are  at  Ellington 
ANGB)  substantial  interaction  among  personnel  is  inevitable. 
Airmen  will  talk  among  themselves  both  within  and  across  workcen¬ 
ters  about  the  AOTS  prototype  -  its  features,  what  they  like  and 
dislike,  etc.  In  cases  where  we  identify  and  use  a  nonequivalent 
control  group,  interactions  among  airmen  from  AOTS  and  co'ntrol 
group  workcenters  will  occur,  and  airmen  may  be  transferred  be¬ 
tween  AOTS  and  control  group  workcenters.  All  of  these  factors 
will  contaminate  the  treatment  groups. 

'nternal  Validity  -  The  MTP  is  vulnerable  to  several  inter¬ 
na,.  validity  concerns.  First,  there  is  always  the  possibility 
that  changes  external  to  the  ACTS  program  could  account  for  any 
differences  from  pre-  to  post-AOTS.  As  an  extreme  example,  the 
F-4s  might  be  replaced  with  another  aircraft.  Vulnerability  to 
such  changes  is  high,  because  of  the  length  of  time  during  which 
the  baseline  and  treatment  take  place  and  because  of  the  high 
priority  which  must  be  accorded  to  operational  considerations  in 
the  workcenters.  Another  problem  is  that  people  themselves 
change  over  time,  irrespective  of  the  AOTS  program  jn  which  they 
may  be  participating.  Any  observed  effects  after  imple  'Station 
of  AOTS  could  be  due  to  natural  maturation  of  the  j  -sonnel 
rather  than  to  AOTS  per  se. 

Treatment  Period  -  Officially,  the  AOTS  program  is  to  be 
implemented  for  a  test  period  (Phase  III)  of  one  year.  However, 
workcenter  personnels- will  require  at  least  a  month  to  "come  up  to 
speed"  in  using  AOTS,  and  data  analysis,  evaluation,  interpreta¬ 
tion,  and  reporting  will  take  at  least  two  months  at  the  end  of 
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the  period.  Thus  the  time  period  during  which  the  AOTS  can  make 
its  impact  on  the  test  results  will  be,  at  most,  nine  months. 
During  this  nine  months,  only  a  few  airmen  will  move  from  entry 
into  a  position  to  full  position  qualification.  Most  airmen  will 
already  be  fully  or  partly  position  qualified  when  the  AOTS 
program  begins,  compounding  the  difficulty  of  demonstrating  the 
effects  of  AOTS  training.  Again,  these  problems  prevent  any 
"pure”  assessment  of  the  effectiveness  of  AOTS. 

Instrumentation  -  Due  to  practical  considerations  (including 
the  need  to  keep  interference  with  normal  Air  Force  operations  at 
a  minimum  while  evaluating  AOTS)  and  costs,  it  will  generally  not 
be  feasible  to  develop  and  use  new  instruments  or  to  design  new 
variables  for  measuring  the  effectiveness  of  AOTS.  Instead,  the 
evaluation  will  have  to  rely  mostly  on  existing  sources  of  data, 
including  variables  currently  measured  in  each  AFS,  as  well  as 
existing  data  bases.  Where  new  instruments  are  developed  (e.g., 
questionnaires  and  structured  interviews) ,  it  will  not  be  pos¬ 
sible  to  pilot  test  them  extensively  to  refine  their  psychometric 
properties  and  ensure  their  reliability  and  validity. 


Samples  -  The  number  of  personnel  in  the  AOTS  and  control 
group  workcenters  at  Bergstrom  AFB  and  Ellington  ANGB  is  another 
limitation.  The  workcenters  are  quite  small  and  are  unequal  in 
size,  ranging  from  l  to  about  50  airmen.  Sample  sizes  for 
analyses  are  further  reduced  below  the  number  in  the  workcenters 
by  fractionation  into  supervisors,  trainers,  and  trainees.  These 
problems  lead  to  e  lack  of  power  in  statistical  hypothesis  test¬ 
ing  and  to  unbalanced ^designs  for  data  analysis. 

External  Validity  -  The  mere  act  of  measuring  personnel  may 
sensitize  them  to  what  is  being  measured  and  to  what  is  expected, 
or  may  motivate  them  to  improve  their  performance.  Data  collec¬ 
tion  will  necessarily  have  to  take  place  in  AOTS  workcenters 
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during  the  baseline  period  before  AOTS  is  implemented,  after  AOTS 
implementation,  and  in  control  groups-  Workcenter  personnel  will 
be  responding  to  questionnaires  and  participating  in  interviews. 
The  reactive  effects  of  these  experiences  can  cloud  attempts  to 
measure  the  impact  of  the  AOTS  program.  Finally,  because  the 
AOTS  prototype  is  being  tested  on  a  small  scale  in  five  AFSs  at 
one  AFB  for  active  and  reserve  and  one  ANGB  for  Guard,  the 
validity  of  generalizing  the  results  of  the  evaluation  to  other 
AFSs  or  to  other  bases  will  be  very  questionable. 
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4 . 6  Management 


4.6.1  Test  Management.  The  responsibilities  of  principal  par¬ 
ticipants  in  the  SLT&E  of  the  AOTS  are  shown  in  Figure  11.  The 
principal  participants  are  as  follows. 

A.  AFHRL 

B.  DAC 


C.  Ball  Systems  Engineering 


Other  participants  to  be  involved  in  the  SLT&E  of  the  AOTS 
in  either  advisory  or  coordinating  roles  include  the  following. 


A. 

HQ  USAF/DPPT 

G. 

HQ  ATC/DPAE 

B. 

HQ  USAF/LEY 

H. 

HQ  AFRES/DPTS 

C. 

AFMPC / DPMRTC  3 

I. 

ANGSC/TET 

D. 

HQ  TAC/DP7  T 
•HQ  TAC/LGMT 

J. 

AFHRL/ LRC 

E. 

HQ  SAC/DPAT 

K. 

AFOTEC 

F. 

HQ  MAC/DPAT 

L. 

BDM 
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4.6.2  AQTS  Program  Management 


A.  Air  Force  Program  Manager — Major  Jack  L.  Blackhurst 

B.  PEM— -Major  Glen  Tanner 

C.  SYSTO — Captain  Frank  Vaccaro 

D.  DAC  Program  Manager — Mr.  D.  Brooks 
4.7  Training  Requirements 

4.7.1  Test  Team  Training.  DAC,  with  AF  members  of  the  test 
team,  will  collect  SLT&E  data  and  will  enter  data  into  the  ACTS 
test  data  base.  Training  times  are  included  within  the  estimates 
in  Appendices  E  and  H.  DAC  will  provide  demonstrations  and  over- 
the-shoulder  training  for  test  team  members  in  the  procedures 
for: 


-  Interviewing 

-  Administering  questionnaires 

-  Collecting  other  data  for  performance  and  suitability 

-  Entering  data  into  the  test  data  base 


4.7.2  SLT&E  Requirements  for  Training  Workcenter  Personnel. 
SLT&E  will  impose  very  few  special  requirements  for  training 

4 

workcenter  personnel.  Principally,  personnex  in  workcenters  will 
need  to  learn  how  to  complete  such  SLT&E  forms  as  weekly  time 
sheets.  The  test  tjaam  will  provide  this  training  as  they  carry 
out  the  test  procedures  provided  by  DAC. 
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4 . 8  Environmental  Impact 
Not  Applicable. 

4 . 9  Safety 

No  special  safety  requirements  are  imposed  as  a  result  of 
SLT&E  activities. 

4 . 10  Security 

All  portions  of  this  test  plan  and  of  the  AOTS  program  are 
unclassified. 

4.11  Release  of  Information 

No  restrictions  except  as  imposed  on  DAC  by  the  AOTS  con¬ 
tract. 

4.12  Intelligence  Threat 
Not  Applicable. 
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ACRONYMS 


AF 

AFB 

AFFM 

AFHRL 

AFMPC 

AFOSP 

AFOTEC 

AFRES 

AFS 

AFSC 

AGB 

AGS 

AMD 

AMU 

ANG 

ANGB 

ANGSC 

AOTS 

APDS 

ATC 

-x  mn 

CAM 

CAMS 

CCN 


Air  Force 
Air  Force  Base 
Air  Force  Form 

Air  Force  Human  Resources  Laboratory 

Air  Force  Military  Personnel  Center 

Air  Force  Office  for  Security  Police 

Air  Force  Operational  Test  and  Evaluation  Center 

Air  Force  Reserves 

Air  Force  Specialty 

Air  Force  Specialty  Code 

Aircraft  Generation  Branch 

Aircraft  Generation  Squadron 

Aerospace  Medical  Division 

Aircraft  Maintenance  Unit 

Air  National  Guard 

Air  National  Guard  Base 

Air  National  Guard  Support  Center 

Advanced  On-the-job  Training  System 

Advanced  Personnel  Data  System 

Air  Training  Command 

Airman  Training  Records 

m 

Criteria  Acquisition  Model 

Consolidated  Aircraft  Maintenance  Squadron 

Contract  Change  Notice 
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CDRL 

CONUS 

CPC 

CPCI 

CRS 

CSG 

CSS 

DAC 

DBA 

DBMS 

DCR 

DESIRE 

DID 

DIR 

DP 

DPMQ 

DTM 

EMS 

FIG 

GPTR 

HQ 

ISD 

1ST 

TTP 

JCN 

JEIM 

JQS 


Contract  Deliverable  Requirements  List 

Continental  United  States 

Computer  Program  Component 

Computer  Program  Configuration  Items 

Component  Repair  Squadron 

Combat  Support  Group 

Combat  Support  Squadron 

Douglas  Aircraft  Company 

Data  Base  Administrator 

Data  Base  Management  System 

Data  Collection  Representative 

Direct  English  Statement  Retrieval 

Data  Item  Description 

Data  Input  Representative 

Directorate  of  Personnel 

Office  Symbol  (Director  of  Personnel  Mgt.  Qualification) 

DAC  Test  Manager 

Equipment  Maintenance  Squadron 

Fighter  Interceptor  Group 

Generic  Position  Task  Requirements 

Headquarters 

Instructional  Systems  Development 
Instructional  Systems  Team 


—  «  •  *  -A-  ^  C*. 


m  _  z  1 


j--1.ajLj1a.n9  nequ jll eiuents 


Job  Control  Number 

Jet  Engine  Intermediate  Maintenance 

Job  Qualification  Standard 
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MAJCOM  Major  Command 


MAT  Maintenance  Training  Office  Symbol 

MDC  Maintenance  Data  Collection 

MILAP  Maintenance  Information  Logically  Analyzed  and  Presented 

MMICS  Maintenance  Management  Information  Control  System 

MOA  Memorandum  Of  Agreement 

MTL  Master  Task  List 

MTP  Master  Test  Plan 

NCO  Noncommissioned  Officer 

OJT  on-the-Job  Training 

OMR  Optical  Mark  Reader 

OPTR  Operational  Position  Task  Requirements 

OR  Orderly  Room 

PCO  Program  Contracting  Officer 

POC  Point  of  Contact 

PE  Personnel  Evaluation 

PMD  Program  Management  Directive 

PCIII  Personnel  Concept  Third  Generation 

RPR  Request  for  Personnel  Research 

SLT&E  System  Level  Test  and  Evaluation 

SP  Security  Police 

SPA  Security  Police  Administration 

SPAS  Security  Police  Automated  System 

SFF  Security  Police  Flight 

SPR  Software  Problem  Report 

SPS  Security  Police  Squadron 

SQL  Structured  Query  Language 
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STS 

Specialty  Training  Standard 

TAC 

Tactical  Air  Command 

T&E 

Test  and  Evaluation 

TFG 

Tactical  Fighter  Group 

TP 

Technical  Paper 

TRW 

Tactical  Reconnaissance 

Wing 

TWG 

Test  Working  Group 

UDF 

Unit  Development  Folder 

USAF 

United  States  Air  Force 

WITS 

Weekly  Inventory  of  Time 

Spent 

WSF 

Weapons  Support  Flight 

WUC 

Work  Unit  Code 
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X 

3 

X 

X 

3.7. IE 

MANAGE  OTHER  TRAINING 

REQUIREMENTS 

1 

- 

4 

X 

5 

X 

X 

3. 7. IF 

MANAGE  LOCAL/NEW  TASKS 

1 

- 

3 

X 

4 

X 

X 

3.7.2 

AIRMAN  TRAINING  MANAGEMENT 

COMPONENT 

1 

X 

2 

X 

3.7.2A 

GENERATE  AIRMAN  TRAINING 

RECORD 

1 

X 

2 

X 

X 

3.7.2B 

DIAGNOSE  TRAINING  NEEDS 

1 

X 

2 

X 

X 

3.7. 2C 

MANAGE  TRAINEE  PROGRESS 

1 

- 

2 

X 

3 

X 

X 
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CROSS  REFERENCE,  TEST  ELEMENTS  TO  CRITICAL  ISSUES  (Cont.) 

EVALUATION  SUBSYSTEM 

SPEC.  TEST  ELEMENT  OBJECTIVE  CRITICAL  ISSUE  # 


.  1 

3.7. 1.1 

3.7. 1.2 

3. 7. 1.4 

3.7. 1.5 

3.7.2 

3. 7. 2.1 

3. 7. 2. 2 

3. 7. 2. 3 

3. 7. 2. 4 

3. 7. 2. 5 

3. 7. 2. 6 

3.7.3 

3. 7. 3.1 

3. 7. 3. 2 

3. 7. 3. 3 


EVALUATION  SUBSYSTEM  1-7 

MANAGE  EVALUATION  INSTRU¬ 
MENTATION  COMPONENT  1 

2 

BEHAVIORAL  OBJECTIVES 

DEVELOPMENT  1 

2 

TEST  ITEM  BANK  1-2 

3 

EVALUATION  MATERIALS 

DEVELOPMENT  1-3 

4 

EVALUATION  INSTRUMENTS 

DELIVERY  1-3 

4 

PERFORMANCE  EVALUATION 

COMPONENT  1 

2-4 

TASK  PERFORMANCE  EVAL¬ 
UATION  1 

2 

REQUISITE  KNOWLEDGE  TEST 

ADMINISTRATION  1-2 

3 

PERFORMANCE  OBSERVATION  1-2 

3 

ACCESS  CONTROL  1-3 

4 

ACCEPT  OFF-LINE  COMPLETION 

RESULTS  3 

2 

COLLECT  PERFORMANCE  EVAL¬ 
UATION  DATA  1 

2 

APPLY  TRAINING  QUALITY 

CONTROL  COMPONENT  \ 

2 

QUALITY  CONTROL  EVALUATION 

EVENT  SELECTION  1 

2 

QUALITY  CONTROL  EVALUATION 

FOLLOw-UP  1-3 

4 

QUALITY  CONTROL  REPORT 

GENERATION  1 

2 


X 


X 

X 

X 

X 

X 

X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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3.7.4  SYSTEM  EVALUATION  COMPON¬ 

ENT 

3. 7. 4.1  UNIT  TRAINING  REPORT  GEN¬ 

ERATION 

3 . 7 . 4 . 2  SYSTEM  EFFECTIVENESS 

REPORT  GENERATION 


1 

2 

1 

2 


1-3 


4 


X 

X 

X 

X 

X 


X 


X 


X 
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INTRODUCTION 


The  following  Test  Plans  reference  sections  of  the  AOTS  B1 
Specifications  (in  the  versions  current  as  of  7  Nov  1986)  and  the 
Software  Test  Plan  (STP)  (Appendix  D  to  this  Master  Test  Plan) . 

Each  Test  Plan  identifies  what  is  being  tested  by  naming  the 
subsystem,  the  paragraph (s)  in  the  associated  Prime  Item  (Bl) 
specification,  and  the  name  of  the  element  (subsystem,  component, 
or  subcomponent) .  The  Test  Objectives  identify  the  functions 
being  tested,  indicating  whether  each  is  an  off  line  (manual)  or 
on  line  (automated)  function  and  giving  a  description  abstracted 
from  the  referenced  specification  paragraphs.  After  the  statement 
of  the  Test  Objectives,  the  evaluation  questions  are  listed.  Each 
evaluation  question  is  keyed  to  one  or  more  of  the  Critical  Issues 
which  are  described  in  MTP  paragraphs  1.5.1  through  1.5.4 
(respectively  Compliance,  Performance,  Suitability,  and 
Acceptance)  of  this  Master  Test  Plan.  For  example,  "Cl  1,  3"  at¬ 
tached  to  an  Evaluation  Question  indicates  the  question  addresses 
Critical  Issues  1  and  3,  Compliance  and  Suitability. 

Each  evaluation  question  is  addressed,  by  number,  under  each 
of  the  following  sections  of  a  Test  Plan: 

Level  of  T&E  Required  -  Whether  the  question  requires  Test 
(T) ,  Inspection  (I),  Analysis  (A),  and/or  Evaluation  (E) - 

Scheduled  Test  Date  -  Time  period  for  test,  referencing 
the  Master  Test  Plan  Schedule  (Appendix  G  of  the  MTP)  . 

Resources/Facilities  Requirements  -  Who  (DAC  staff,  DAC/IST 
experts,  DAC/AF  inspectors,  1ST)  and  what  facilities  are 
involved  in  the  test. 

Evaluation  Instruments  Requirements  -  Inspection  Review  Form, 
Problem  Report,  or  User  Survey  (see  samples  in  Appendix  B) . 

Baseline  Data  Collection  Requirements  -  Not  applicable  for 
MTP  Part  1  testing. 

Test  Data  Collection  Requirements  -  Sources  of  test  data. 

Data  Analysis  Requirements  -  General  type  of  analysis  to  be 
performed. 

Test  Criteria  Goals  -  Intended  goals  of  the  testing. 

During  alpha  and  readiness  testing,  inputs  and  expected 
results  will  be  as  described  in  the  Software  Test  Plan.  During 
beta  testing,  inputs  will  be  real  data  encountered  in  the  course 
of  implementing  and  using  the  prototype  AOTS.  Expected  results 
are  stated  in  the  Test  Criteria  Goals  for  each  Test  Plan. 

"Expert  judgment"  is  judgment  by  one  or  more  subject  matter 
experts  (designated  as  described  in  Appendix  B)  that  the  product 
observed  or  evidence  gathered  demonstrates  compliance  to 
Specification  requirements. 

"Go-no  go"  is  the  determination,  under  the  procedures  of  the 
Software  Tent  Plan,  that  a  software  product  is  ready  to  advance  to 
the  next  stage  of  testing  or  development  ('go')  or  needs  rework 
before  advancing  ('no  go'). 
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SUBSYSTEM: 


MANAGEMENT 


TEST  PLAN 


B-jl  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7 

TEST  ELEMENT:  Management  Subsystem 

TEST  OBJECTIVE (S ) : 

1.  The  Management  Subsystem  shall  be  compatible  and  interactive 
within  its  own  components. 

2.  The  Management  Subsystem  shall  provide  the  capability  for  the 
user  to  identify  all  AFS  training  required  to  achieve  position 
qualification. 

3.  The  Management  Subsystem  shall  provide  the  capability  for  the 
user  to  manage  and  record  training  progress  toward  task 
proficiency  and  position  qualification. 

4.  The  Management  Subsystem  shall  provide  the  capability  for  the 
user  to  schedule  training  efficiently  and  effectively. 

5.  The  Management  Subsystem  shall  maintain 
proficiency/performance  data  at  the  job  task  level. 

6.  The  Management  Subsystem  shall  be  compatible  and  interactive 
with  the  other  AOTS  subsystems. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Do  the  subcomponents  interact  effectively? 

CI-1 

2-6.  Do  the  on-line  capabilities,  manual  tools,  and  procedures 
established  in  the  Management  Subsystem  satisfy  the  AF  require¬ 
ments  for  AOTS? 

CI-1 

7.  Dees  the  subsystem  interact  effectively  with  other  AOTS 
subsystems? 

LEVEL  OF  T&E  REQUIRED: 

i/  ?•  T  see  Software  Test  Plan  for  details  on  integration  test¬ 
ing  within  the  Management  Subsystem,  and  between  Management  Sub¬ 
system  and  other  AOTS  Subsystems,  and  external  interfaces  with  Air 
Force. 


2-6.  E  expert  judgment  based  on  summary  data  from  lower  level 
evaluations. 
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SCHEDULED  TEST  DATE: 


1,  7.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

2-6.  Review  completed  within  10  days  following  development;  see 
schedule,  Appendix  G. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1,  7.  DAC  staff  utilizing  Air  Force  facilities/equipment/ dummy 

and/or  real  data. 

2-6.  DAC/AF  experts  in  Buildings  1808  and  428. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1,  7.  Software  Alpha  Test  procedures. 

2-6.  Inspector  Review  Form. 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1,  7.  Software  Alpha  Test  procedures. 

2-6.  DAC/AF  expert(s)  review  of  plans,  products,  and  records. 

DATA  ANALYSIS  REQUIREMENTS: 

1,  7.  Analysis  of  Alpha  Test  results. 

2-6.  Descriptive  judgments. 

TEST  CRITERIA  GOALS: 

1,  7.  Go/No  Go  Status. 

2-6.  Expert  judgment  that  product (s)  satisfies  requirements. 
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TEST  PLAN 


SUBSYSTEM:  MANAGEMENT _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.1 _ 

TEST  ELEMENT :  Training  Requirements  Management  Component _ 

TEST  OBJECTIVE (S) : 

1.  The  component  shall  provide  valid  and  reliable  methodologies 
and  mechanisms  for  generating  a  master  task  list,  obtaining  AFS 
performance  and  proficiency  data,  creating  evaluation  and  training 
requirements,  generating  position  training  requirements,  and 
monitoring  other  necessary  training  requirements. 

2 .  The  component  shall  be  compatible  and  interactive  within  and 
among  its  own  subcomponents. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-4 

1.  Does  the  Training  Requirements  Management  Component  provide 
the  user  with  acceptable  methods  and  tools  needed  to  identify  and 
manage  airmen  training  requirements? 

CI-1 

2.  Do  the  subcomponents  interact  effectively? 

LEVEL  OF  T&E  REQUIRED: 

1.  E  user  Acceptance  Survey  Results  -  Component  level. 

2.  T  see  Software  Test  Plan  for  details  on  interactive  testing 
at  the  component  level. 

SCHEDULED  TEST  DATE: 

1.  Surveys  administered  during  Beta  Tests. 

2.  Alpha  Test  immediately  following  software  development?  see 
schedule,  Appendix  G. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1.  AF  Workstation/AF  Workgroup. 

2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 


A- 9 


EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  User  surveys  (see  sample  for  software  products) . 

2.  Software  Alpha  Test  procedures. 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  AF  reactions  to  use  of  products. 

2.  See  Software  Test  Plan. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Descriptive  individual  item  and  composite  score  data  analysis. 

2.  Analysis  of  Alpha  Test  results. 

TEST  CRITERIA  GOALS: 

1.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 

2.  Go/No  Go  Status. 
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TEST  PLAN 


SUBSYSTEM:  MANAGEMENT 


B±  SPEC  REQUIREMENT  REFERENCE  NO.:  3 . 7 . 1A _ 

TEST  ELEMENT:  Develop  Master  Task  List _ 

TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  access  Air  Force  data  and  to  generate  and  review  a 
Tentative  Master  Task  List  (TMTL) . 

2.  Off-line.  The  subcomponent  shall  provide  guidelines  and  coor¬ 
dination  procedures  for  validating  the  master  task  list  of  all 
performed  within  an  AFS . 

3.  On-line.  The  subcomponent  chall  provide  the  capability  for 
the  user  to  generate  and  review  a  validated  final  Master  Task  List 
( FMTL) .  Average  time  to  search  for  and  display  an  isolated  task 
shall  not  exceed  sixty  seconds. 

4.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 

1.1. 1.1.1  -  1.1. 1.1. 6  met? 

CI-1 

2.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 

1.1. 1.2.1  -  1.1. 1.2. 4  met? 

3.7.1A  cont. 

CI-1 

3.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 

1.1. 1.3.1  -  l.i. 1.3. 6  met? 


CI-l 

4a.  Does  the  associated  hardware  support  the  software  products? 


CI-4 

4b •  Do  ths  ussrs  of 
satisfactory? 


~  —  — 


tools , 


dnu  procedures  rate  each  as 


LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan  for  further  details 

2.  I  expert  judgment 

3.  T  see  Software  Test  Plan  for  further  details 

4a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
4b.  A  user  surveys 

SCHEDULE  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development; 
see  phased-in  schedule 

2.  Review  completed  within  10  days  following  development; 
see  phased-in  schedule 

3.  Alpha  Test  immediately  following  software  development; 
see  phased-in  schedule 

4a.  Part  of  Alpha  Test  procedures 

4b.  Surveys  administered  during  Beta  Tests  for  1,2,3 
RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data 

2.  DAC/AF  experts  in  Buildings  1808  and  428 

3.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data 

4a.  DAC/AF  inspectors  for  review  of  software  problem  report 
4b.  AF  Workstation/AF  Workgroup 
EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures 

2 .  Inspector  Review  Form 

3 .  Software  Alpha  Test  procedures 
4a.  Software  Problem  Report 

4b.  Users  surveys  (see  sample  for  software  products) 
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BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details 

2.  DAC/AF  expert (s)  review  of  off-line  plans  and  products 

3.  See  Software  Test  Plan  for  further  details 
4a.  See  Software  Test  Plan  for  further  details 
4b.  AF  reactions  to  use  of  products 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results 

2.  Descriptive  judgments 

3 .  Analysis  of  Alpha  Test  results 
4a.  Analysis  of  Alpha  Test  results 

4b.  Descriptive  individual  item  and  composite  score  data  analysis 
TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status 

2.  Expert  judgment  that  product (s)  satisfies  requirements 

3 .  Go/No  Go  Status 
4a.  Go/No  Go  Status 

4b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training 
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TEST  PLAN 


SUBSYSTEM:  MANAGEMENT _ _ _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO. :  3 . 7 . IB _ 

TEST  ELEMENT:  Maintain  AFS  Task  Performance  and  Proficiency  Data 


TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  identify  sources  of  task  performance  and  proficiency 
data,  cross  reference  sources  and  tasks,  catalogue  task  perfor¬ 
mance  and  proficiency  source  identifications  for  easy  assess. 

2.  on-line.  Following  the  manual  review  of  new  publications  or 
changes  to  existing  publications  for  update  requirements,  the  sub¬ 
component  shall  provide  the  capability  for  the  user  to  store  new 
tasks  and  flag  follow-up  requirements,  and  flag  tasks,  objectives, 
evaluation  instruments,  and  training  material  identities  for 
review. 

3.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary, 

Processes  1.1. 2.1  -  1. 1.2.3  met? 

CI-1 

2.  Are  the  requirements  as  specified  in  Data  Dictionary, 

Processes  1.1.2. 4  -  1. 1.2.6  met? 

CI-1 

3.  a.  Does  the  associated  hardware  support  the  software 

products? 

CI-4 

b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each 
as  satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.2.  T  see  Software  Test  Plan 

3a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
3b.  A  user  surveys 
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SCHEDULED  TEST  DATE: 


1,  2.  Alpha  Test  immediately  following  software  development?  see 
schedule,  Appendix  G. 

3a.  Software  Problem  Report  results. 

3b.  Surveys  administered  during  Beta  Tests. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1,  2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 

and/or  real  data. 

3a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

3b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1,  2.  Software  Alpha  Test  procedures. 

3a.  Software  Problem  Report. 

3b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1,  2.  See  Software  Test  Plan  for  details. 

3a.  See  Software  Test  Plan  for  details. 

3b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1,  2.  Analysis  of  Alpha  Test  results. 

3a.  Analysis  of  Alpha  Test  results. 

3b. ^  Descriptive  individual  item  and  composite  score  data 
analysis . 
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TEST  CRITERIA  GOALS: 


1,  2.  Go/No  Go  Status. 

3a,  Go/No  Go  Status. 

3b.  Majority  of  users  rate  product  satisfactory; 
specific  comments  for  possible  revisions  or  additional 


review 

training. 


TEST  PLAN 


SUBSYSTEM:  MANAGEMENT - - - - 

B-l  SPEC  REQUIREMENT  REFERENCE  NO.:  _ 3. 7.  ID _ 

TEST  ELEMENT:  Manage  Position  Training  Requirements _ 

TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  to 
generate  and  prioritize  Generic  Position  Task  Requirements  and 
Operational  Position  Task  Requirements. 

2.  Off-line.  Verification  of  position  types  and  verification  of 
duty  position  titles  and  their  matches  with  duty  position  types 
shall  be  accomplished  by  the  Air  Force  through  coordination  with 
Work  Center  supervisors,  and  is  not  subject  to  evaluation  under 
the  MTP. 

3.  The  plans,  tools,  and  procedures  shall  be  user  friendly  and 
supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Processes 
1.1. 4.1  and  1.1. 4. 2  met? 

CI-1 

3a.  Does  the  associated  hardware  support  the  software  products? 


CI-4 

3b.  Do  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

3a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
3b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development?  see 
schedule,  Appendix  G, 

3a.  Part  of  Alpha  Test  procedures. 

3b.  Surveys  administered  during  Beta  Tests. 
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RESOURCES/ FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/ dummy 
and/or  real  data. 

3a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

3b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Softv/are  Alpha  Test  procedures. 

3a.  Softv/are  Problem  Report. 

3b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

3a.  See  Software  Test  Plan  for  further  details. 

3b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

3a.  Analysis  of  Alpha  Test  results. 

3b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

3a.  Go/No  Go  Status. 

3b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training 
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TEST  PLAN 


SUBSYSTEM:  MANAGEMENT _ _ 

B-l  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.  IE _ 

TEST  ELEMENT:  Manage  Other  Training  Requirements _ 

TEST  OBJECTIVE (S ) : 

1.  Off-line.  The  subcomponent,  shall  (or  the  user  shall)  manually 

research  the  appropriate  document  or  source  and  identify:  ap¬ 

proved  Ancillary  Training  Requirements,  approved  Additional  Duties 
Training  Requirements,  Contingency  Task  Requirements  and  Career 
Development  Course  training  requirements. 

2.  Off-line.  The  subcomponent  shall  establish  coordination  plans 
and  procedures  to  ensure  approved  revisions  to  policy  related  to 
other  training  requirements  are  identified  in  advance  of  implemen¬ 
tation  to  permit  proper  planning  and  execution  of  system  changes, 
as  required. 

3.  Off-line.  The  subcomponent  shall  develop  algorithms  to  select 
training  requirements  co  be  managed  in  the  Airman  Training  Manage¬ 
ment  component,  and  to  support  the  identification  of  potential 
position  task  training  requirements  after  OPTRs  have  been  compared 
to  the  ATR. 

4.  On-line.  The  subcomponent  shall  provide  the  capability  to  en¬ 
ter  and  update  training  requirements  and  frequencies  of  training 
as  needed;  match  requirements  to  organizational  levels,  AFSs,  or 
event  or  condition  for  which  they  are  prescribed;  assign  iden¬ 
tification  codes  to  each  course  or  training  requirement;  and  to 
store  both  requirements  and  selection  algorithms  in  the  system. 

5.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1,  2.  Are  the  manual  requirements  as  specified  in  Data  Dictionary 
Processes  1.1.5. 1  -  1. 1.5.6  met? 

CI-1 

3.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 
1 . 1 . 5 . 7  met? 

CI-1 

4.  Are  the  automated  requirements  as  specified  in  Data  Dictionary 
Processes  1. 1.5.1  -  1.1. 5. 6,  and  1.1. 5. 8  met? 
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CI-1 

5a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

5b.  Do  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1,  2,  3.  I  expert  judgment 

4 .  T  see  Software  Test  Plan 

5a.  I  Software  Problem  Report  results,  see  Software  Test  Plan 
5b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1,  2,  3.  Review  completed  within  10  days  following  development? 

see  schedule,  Appendix  G. 

4.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

5a.  Part  of  Alpha  Test  procedures. 

5b.  Surveys  administered  during  Beta  Tests. 

RESOURCES/FACILITIES  REQUIREMENTS: 

1,  2,  3.  DAC/AF  experts  in  Buildings  1808  and  428. 

4.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

5a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

5b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1,  2,  3.  Inspector  Review  Form. 

4.  Software  Alpha  Test  procedures. 

5a.  Software  Problem  Report. 

5b.  User  surveys  (see  sample  for  software  products) . 
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BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1,  2,  3.  DAC/AF  expert (s)  review  of  off-line  plans  and  products. 
4.  See  Software  Test  Plan  for  further  details. 

5a.  See  Software  Test  Plan  for  further  details. 

5b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1,  2,  3.  Expert  judgment  that  product(s)  satisfies  requirements. 
4.  Analysis  of  Alpha  Test  results. 

5a.  Analysis  of  Alpha  Test  results. 

5b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1,  2,  3.  Expert  judgment  that  product(s)  satisfies  reg”'  -ements. 
4.  Go/No  Go  Status. 

5a.  Go/No  Go  Status. 

5b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  MANAGEMENT _ 

B-jl  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.  IF _ 

TEST  ELEMENT:  Manage  Local/New  Tasks _ 

TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  add  new/local  AFS  tasks  to  the  OPTR  task  list. 

2.  Off-line.  The  subcomponent  shall  provide  hard  copies  of  all 
tasks  that  are  being  performed  within  a  specified  AFS  but  were  not 
contained  in  occupational  survey  data  for  that  AFS  to  the  AFOMC. 

3.  Off-line.  The  subcomponent  shall  make  decisions  concerning 
whether  or  not  to  include  Local  or  New  Tasks  on  applicable  MTLs 
based  on  manually  supplied  data  indicating  how  widespread  the 
requirements  are  for  performing  the  tasks. 

NOTE:  Following  actual  implementation  of  the  prototype  AOTS,  the 

capability  shall  exist  for  this  task  use  data  to  be  collected  and 
reviewed  on-line. 

4.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  automated  and  automatic  requirements  as  specified  in 
Data  Dictionary  Processes  1.1. 5.1  -  1.1. 6.9  met? 

CI-1 

2.  Are  the  manual  requirements  as  specified  in  Data  Dictionary 
Process  1.1. 6. 5  met? 

CI-1 

3.  Are  the  manual  requirements  as  specified  in  Data  Dictionary 
Process  1.1. 6. 9  met? 

CI-1 

4a.  Does  the  associated  hardware  support  the  software  products? 


CI-4 

4b.  do  users  of  the  plans,  tools,  and  CI-1  procedures  rate  each  as 
satisfactory? 


LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2.  I  expert  judgment 

3.  I  expert  judgment 

4a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
4b.  A  user  survey 

SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

2,  3.  Review  completed  within  10  days  following  development;  see 
schedule.  Appendix  G. 

4a.  Part  of  Alpha  Test  procedures. 

4b.  Surveys  administered  during  Beta  Tests. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

2,  3.  DAC/AF  experts  in  Buildings  1808  and  428. 

4a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

4b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2,  3.  Inspector  Review  Fcrr . 

4a.  Software  Problem  Report. 

4b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 


N/A 


TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2,  3.  DAC/AF  expert (s)  review  of  off-line  plans  and  products. 

4a.  See  Software  Test  Plan  for  further  details. 

4b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2,  3.  Descriptive  judgments. 

4a.  Analysis  of  Alpha  Test  results. 

4b.  Descriptive  individual  item  and  composite  score  data 
analysis . 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2,  3.  Expert  judgment  that  product (s)  satisfies  requirements. 

4a.  Go/No  Go  Status. 

4b.  Majority  of  users  rate  product  satisfactory;  review  oi 
specific  comments  for  possible  revisions  or  additional  training. 


TEST  PLAN 


SUBSYSTEM:  MANAGEMENT _ 

B-jl  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.2 _ 

TEST  ELEMENT :  Airman  Training  Management  Component _ 

TEST  OBJECTIVE (S) : 

1.  The  component  shall  provide  valid  and  reliable  methodologies 
and  mechanisms  for  creating  automated  Airman  Training  Records  on 
line,  identifying  individual  training  requirements,  and  managing 
and  tracking  training  requirements  through  certification. 

2.  The  component  shall  be  compatible  and  interactive  within  and 
among  its  own  subcomponents. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-4 

1.  Does  the  Airman  Training  Management  Component  provide  the  user 
with  acceptable  methods  and  tools  needed  to  track  and  manage  in¬ 
dividual  airman  training  requirements  through  certification? 

CI-i 

2.  Do  the  subcomponents  interact  effectively? 


LEVEL  OF  T&E  REQUIRED: 

1.  E  User  Acceptance  Survey  results  -  component  level 

2.  T  see  Software  Test  Plan  for  details  on  interactive  testing 
at  the  component  level 

SCHEDULED  TEST  DATE: 

1.  Surveys  administered  during  Beta  Tests  for  1,  2,  3. 

2.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1.  AF  Workstation/AF  Workgroup. 

2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 


EVALUATION  INSTRUMENTS  REQUIREMENTS: 


lc  User  surveys  (see  sample  for  software  products) . 

2.  Software  Alpha  Test  procedures. 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  AF  reactions  to  use  of  products. 

2.  See  Software  Test  Plan  for  further  details. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Descriptive  individual  item  and  composite  score  data  analysis. 

2.  Analysis  of  Alpha  Test  results. 

TEST  CRITERIA  GOALS: 

1.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 

2.  Go/Nc  Go  Status. 
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TEST  PLAN 


SUBSYSTEM:  MANAGEMENT _ 

B1  SPEC  REQUIREMENT  REFERENCE  HO.,:  3.7.2A _ 

TEST  ELEMENT:  Generate  Airman  Training  Record _ 

TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  create,  access,  and  review  new  data  in  a  permanent 
training  record  of  all  training  received,  current  training  status, 
position  qualification  and  task  certification  history,  and  task 
trainer  qualifications  records. 

2,  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  1.2. 1.1- 
1.2. 1.2  met? 

CI-1 

2a.  Does  the  associated  hardware  support  the  software  products? 


CI-4 

2b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  eacr.  as 
satisfactory? 

LEVEL  OF  T&E  RLQUIRED; 

1.  T  see  Software  Test  Plan 

2a.  I  Software  Problem  Report  results,  see  Software  Test  Plar 
2b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

2a.  Part  of  Alpha  Test  procedures. 

2b.  Surveys  administered  during  Beta  Tests. 
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RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  f acilit ies/equipment/dummy 
and/or  real  data. 

2a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

2b.  AF  Workstation/AF  Workgroup, 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2a.  Software  Problem  Report. 

2b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2a.  See  Software  Test  Plan  for  further  details. 

2b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2a.  Analysis  of  Alpha  Test  results. 

2b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2a.  Go/No  Go  Status. 

2b.  Majority  of  users  rate  product  satisf actorv :  review  of 
specific  comments  tor  possible  revisions  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  MANAGEMENT _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.2B _ 

TEST  ELEMENT :  Diagnose  Training  Needs _ ; _ 

TEST  OBJECTIVE (S )  : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  review  position  qualifications  and  qualification 
assessments  and  to  provide  accurate  information  of  the  training 
required  for  an  airman  to  become  position  qualified. 

2.  The  subcomponent  plans,  procedures,  and  tasks  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary, 
Processes  1.2. 2.1  -  1.2. 2. 7  met? 

CI-1 

2a.  Does  associated  hardware  support  the  software  products? 

CI-4 

2b.  Do  the  users  of  the  plans,  tools  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
2b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule.  Appendix  G. 

2a.  Part  of  Alpha  Test  procedures. 

2b.  Surveys  administered  during  Beta  Tests. 


RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilif ies/equipment/dummy 
and/or  real  data. 

2a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

2b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2a.  Software  Problem  Report. 

2b.  Us  r  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2a.  See  Software  Test  Plan  for  further  details. 

2b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2a.  Analysis  of  Alpha  Test  results. 

2b.  Descriptive  indivioual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2a.  Go/No  Go  Status. 

2b.  Majority  of  users  rate  product  satisfactory?  review  of 
specific  comments  for  possible  revisions  or  addit  ional  training. 
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TEST  PLAN 


SUBSYSTEM:  MANAGEMENT 


B-j^  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.2C 
TEST  ELEMENT:  Manage  Trainee  Progress 


TEST  OBJECTIVE (S) : 


1.  On-line, 
the  user  to 
requirements, 
reports. 


The  subcomponent  shall  provide  the  capability  for 
consolidate  and  coordinate  training/evaluation 
trainee  progress,  and  generate  training  progress 


2.  Off-line.  The  subcomponent  shall  manually  pass  printed  lists 
of  training  and  evaluation  requirements  which  are  scheduled  and 
conducted  outside  of  AOTS  to  the  applicable  agencies. 


3.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 


EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  automated  and  automatic  requirements  as  specified  in 
Data  Dictionary  Processes  1.2. 3.1  -  1.2. 3. 5  met? 

CI-1 

2.  Are  the  manual  requirements  as  specified  in  Data  Dictionary 
Process  1.2. 3. 2  met? 


CI-1 

3a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

3b.  Do  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 


1. 

T 

see  Software  Test  Plan 

2. 

I 

expert  judgment 

3  a . 

I 

software  Problem  Repcr 

3b. 

A 

user  survey 

see  Software  Test  Plan 
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SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule.  Appendix  G. 

2.  Review  completed  within  10  days  following  development;  see 
schedule,  Appendix  G. 

3a.  Part  of  Alpha  Test  procedures. 

3b.  Surveys  administered  during  Beta  Tests. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

2.  DAC/AF  experts  in  Buildings  1808  and  428. 

3a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

3b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2.  Inspector  Review  Form. 

3a.  Softwax'e  Problem  Report. 

3b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2.  DAC/AF  expert (s)  review  of  off-line  plans  and  products. 

3a.  See  Software  Test  Plan  for  further  details. 

3b.  AF  reactions  to  use  of  products. 


DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2.  Descriptive  judgments. 

3a.  Analysis  of  Alpha  Test  results. 

3b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2.  Expert  judgment  that  product (s)  satisfies  requirements. 

3a.  Go/No  Go  Status. 

3b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

B1  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7 

TEST  ELEMENT:  Evaluation  Subsystem 
TEST  OBJECTIVE (S) : 

1.  The  Evaluation  Subsystem  shall  be  compatible  and  interactive 
within  its  own  components. 

2.  The  Evaluation  Subsystem  shall  provide  the  capability  for  the 
user  to  effectively  develop  and  administer  airman  performance 
measures. 

3.  The  Evaluation  Subsystem  shall  provide  the  capability  for  the 
user  to  develop  adequate  task  measurement  criteria  and  standards 
for  individual  airman  proficiency  evaluation  task  certifications. 

4.  The  Evaluation  Subsystem  shall  provide  the  capability  for  the 
user  to  efficiently  collect  and  report  system/program  evaluation 
data. 

5.  The  Evaluation  Subsystem  shall  provide  the  capability  for  the 
user  to  apply  effective  methods  of  training  quality  control. 

6.  The  Evaluation  Subsystem  shall  ensure  proper  security  controls 
over  evaluation  materials  and  evaluation  results. 

7 .  The  Evaluation  Subsystem  shall  be  compatible  and  interactive 
with  the  other  AOTS  subsystems. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Do  the  subcomponents  interact  effectively? 


CI-1 

2-6.  Do  the  on-line  capabilities,  manual  tools,  and  procedures 
established  in  the  Evaluation  Subsystem  satisfy  the  AF  require¬ 
ments  for  AOTS? 

CI-1 

7.  Does  the  subsystem  interact  effectively  with  other  AOTS 
subsystems? 


A-34 


LEVEL  OF  T&E  REQUIRED: 

1,  7.  T  see  Software  Test  Plan  for  details  on  integration  test¬ 
ing  within  the  Evaluation  Subsystem,  and  between  the  Evaluation 
Subsystem  and  other  AOTS  subsystems. 

2-6.  E  expert  judgment  based  on  summary  data  from  lower  lev  il 
evaluations. 

SCHEDULED  TEST  DATE: 

1,  7.  Alpha  Test  immediately  following  software  development;  see 
schedule ,  Appendix  G . 

2-6.  Review  completed  within  10  days  following  development;  see 
schedule,  Appendix  G. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1,  7.  DAC  staff  utilizing  Air  Force  facilities/equipment/ dummy 

and/or  real  data. 

2-6.  DAC/AF  experts  in  Buildings  1808  and  428. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1,  7.  Software  Alpha  Test  procedures. 

2-6.  Inspector  Review  Form. 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1,  7.  See  Software  Test  Plan  for  further  details. 

2-6.  DAC/AF  expert (s)  review  of  off-line  plans  and  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1,  7.  Analysis  of  Alpha  Test  results. 

2-6.  Descriptive  judgments. 

TEST  CRITERIA  GOALS: 

1,  7.  Go/No  Go  Status. 

2-6.  Expert  judgment , that  produqt(s)  satisfies  requirements. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ _ _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3 . 7_.  1 _ 

TEST  ELEMENT:  Manage  Evaluation  Instrumentation  Component _ 

TEST  OBJECTIVE (S ) : 

1.  The  component  shall  provide  valid  and  reliable  methodologies 
and  mechanisms  for  producing,  maintaining,  controlling,  and  manag¬ 
ing  the  evaluation  instruments  required  for  AOTS. 

2.  The  component  shall  be  compatible  and  interactive  within  its 
own  subcomponents. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-4 

1.  Does  the  Manage  Evaluation  Instrumentation  Component  provide 
the  user  with  acceptable  methods  and  tools  needed  to  prepare  and 
manage  evaluation  instruments? 

CI-1 

2.  Do  the  subcomponents  interact  effectively? 

LEVEL  OF  T&E  REQUIRED: 

1.  E  User  Acceptance  Survey  Results  -  component  level 

2.  T  see  Software  Test  Plan  for  details  on  interactive  tests  at 
the  component  level 

SCHEDULED  TEST  DATE: 

1.  Surveys  administered  during  Beta  Tests. 

2.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1.  AF  Workstation/AF  Workgroup. 

2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  User  surveys  (see  sample  for  software  products). 

2.  Software  Alpha  Test  procedures. 
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BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  AF  reactions  to  use  of  products. 

2.  See  Software  Test  Plan  for  further  details. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Descriptive  individual  item  and  composite  score  data  analysis. 

2.  Analysis  of  Alpha  Test  results. 

TEST  CRITERIA  GOALS: 

1.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 

2.  Go/No  Go  Status. 


A-37 


TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

B-jl  SPEC  REQUIREMENT  REFERENCE  NO-  :  3.7. 1.1 _ 

TEST  ELEMENT:  Behavioral  Objectives  Development _ 

TEST  OBJECTIVE (S ) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  access  tasks  and  develop  behavioral  objectives  for  a 
whole  task  and  each  of  its  subtasks,  or  to  revise  existing  objec¬ 
tives. 

2.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 
2.1.1  met? 

CI-1 

2a.  Does  the  associated  hardware  support  the  software  products? 
Cl-4 

2b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan  for  the  details 

2a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
2b.  A  user  survey 
SCHEDULED  TEST  DATE: 


1.  Alpha  Test  immediately  following  software  development;  see 
phased-in  schedule. 

2a.  Part  of  Alpha  Test  procedures. 

2b.  Survey  administered  during  Beta  Test  for  I. 
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RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/ dummy 
and/or  real  data. 

2a.  DAC/AF  inspectors  for  review  of  software  problem  reports. 

2b.  AF  Workstation/AF  Workgroup 
EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  Procedures 
2a.  Software  Problem  Report 

2b.  user  survey  (see  sample  for  software  products) 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  detail 
2a.  See  Software  Test  Plan  for  further  detail 
2b.  AF  reactions  to  use  of  product 
DATA  ANALYSIS  REQUIREMENTS : 

1.  Analysis  of  Alpha  Test  results 
2a.  Analysis  of  Alpha  Test  results 

2b.  Descriptive  individual  item  and  composite  score  data  analysis 
TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status 
2a.  Go/No  Go  Status 

2b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  of  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3. 7. 1.2 _ 

TEST  ELEMENT :  Test  Item  Bank _ 

TEST  OBJECTIVE (S ) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  generate  (develop)  individual  knowledge  test  item  from 
the  behavioral  objectives,  revise,  and  store  them  in  an  item  bank 
store;  conduct  catalog  search  for  knowledge  items,  performance 
measures,  and  oral  test  guides;  select  items  and/or  alternate  test 
by  individual  objectives  and  sets  of  objectives;  select  perfor¬ 
mance  measurement  instruments  and  oral  guides  by  job  task  and/or 
subtask;  set  test  parameters  for  knowledge  tests;  sequence 
measurement  items  and  distractors,  cross  reference  items  to 
products;  and  conduct  test  item  and  performance  measure  analyses. 

2.  Off-line.  The  subcomponent  shall  provide  the  validation 
logic,  plans  and  procedures  for  the  user  to  validate  evaluation 
materials. 

3.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 
2. 1.2.1  -  2. 1.2. 9  met? 

CI-1 

2.  Are  requirements  as  specified  -  Data  Dictionarv  Process 
2.1.2.10  met? 

CI-1 

3a.  Does  the  associated  hardware  support  tie  software  products? 


CI-4 

3b.  Do  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory  or  higher? 
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LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2.  I  expert  judgment 

3a.  I  Software  problem  Report  results  -  see  Software  Test  Plan 
3b.  A  user  survey 

SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development?  see 
phased-in  schedule 

2.  Review  completed  within  10  days  following  development;  see 
phased-in  schedule 

3a.  Part  of  Alpha  Test  procedures 

3b.  Surveys  administered  during  Beta  Test  for  1  and  2 
RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/ dummy 
and/or  real  data 

2.  DAC/AF  experts  in  Buildings  1808  and  428 

3a.  DAC/AF  inspectors  for  review  of  software  problem  report 
3b.  AF  Works tat ion/ AF  Group 
EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  Procedures 

2.  Inspector  Review  form 
3a.  Software  Problem  Report 
3b.  user  surveys 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 
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TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details 

2.  DAC/AF  expert (s)  review  of  off-line  plans  and  products 
3a.  See  Software  Test  Plans  for  further  detail 

3b.  AF  reactions  to  use  of  products 
DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  Results 

2.  Descriptive  judgments 

3a.  Analysis  of  Alpha  Test  results 
3b.  Descriptive  individual  item 
TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status 

2.  Expert  judgment  that  product (s)  satisfies  requirements 
3a.  Go/No  Go  Status 

3b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION 


B1  SPEC  REQUIREMENT  REFERENCE  NO.:  3. 7. 1.4 _ 

TEST  ELEMENT:  Evaluation  Materials  Development _ 

TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  develop  and  revise  modular  evaluation  materials,  in 
accordance  with  ISD  practices. 

2.  Off-line.  The  subcomponent  shall  provide  guidelines  and  pro¬ 
cedures  for  validation  of  evaluation  modules  against  a  criterion 
of  job  performance  in  an  operational  setting. 

3.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  identify  changes  in  existing  task  evaluation  material, 
and  revise  as  necessary. 

4.  The  subcomponent  plans,  tools  and  procedures  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 

2.1.4  met? 

CI-1 

2.  Are  the  requirements  as  specified  in  Data  Dictionary  Processes 
2. 1.4. 1.1  -  2. 1.4. 1.8  met? 

CI-1 

3.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 

2 . 1 . 4 . 2 . 4  met? 

CI-1 

4a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

4b.  Do  the  users  of  the  plans,  tools  and  procedures  rate  each  as 
satisfactory? 
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LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2.  I  expert  judgment 

3.  T  see  Software  Test  Plan 

4a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
4b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  0. 

2.  Review  completed  within  10  days  following  development;  see 
schedule,  Appendix  G. 

3.  Alpha  Test  immediately  following  software  development;  see 
schedule.  Appendix  G. 

4a.  Part  of  Alpha  Test  procedures. 

4b.  Surveys  administered  during  Beta  Tests  for  1,  2,  3. 
RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

2.  DAC/AF  experts  in  Buildings  1808  and  428. 

3.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

4a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

4b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2.  Inspector  Review  Form. 

3.  Software  Alpha  Test  PaTOCCuUl’cS 
4a.  Software  Problem  Report. 

4b.  User  surveys  (see  sample  for  software  products) . 
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BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2.  DAC/AF  expert (s)  review  of  off-line  plans  and  products. 

3.  See  Software  Test  Plan  for  further  details. 

4a.  See  Software  Test  Plan  for  further  details. 

4b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2.  Descriptive  judgments. 

3.  Analysis  cf  Alpha  Test  results. 

4a.  Analysis  of  Alpha  Test  results. 

4b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2.  Expert  judgment  that  product (s)  satisfies  requirements. 

3.  Go/No  Go  Status. 

4a.  Go/No  Go  Status. 

4b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 


TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3. 7. 1.5 _ 

TEST  ELEMENT:  Evaluation  Instruments  Delivery _ 

TEST  OBJECTIVE (S ) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  control  access  to  evaluation  materials  on-line  by 
granting  individuals  access  to  various  Editors  and  by  granting 
various  permissions  to  functions  within  each  Editor;  and  to  con¬ 
trol  the  receipt,  storage,  and  delivery  of  all  evaluation 
materials  during  on-line  test  presentations, 

2.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  control  access  to  printed  evaluation  materials  (off¬ 
line)  by  allowing  the  user  to  enter  a  delivery  and  return  notice 
containing  name,  SSAN,  date,  and  time  of  delivery/return,  and  a 
materials  destroyed  notice  when  applicable. 

NOTE:  Control  of  the  actual  printed  evaluation  materials  is  a 

manual  process  to  be  developed  by  the  Air  Force  and,  therefore, 
not  subject  to  evaluation  under  the  MTP. 

3.  Off-line.  The  subcomponent  shall  provide  the  user  with 
guidelines  for  off-line  procedures  for  storage,  distribution,  and 
collection  of  evaluation  materials  and  devices  for  job  site  task 
performance  evaluations. 

4.  The  subcomponent  plans,  tools,  and  procedures  shall  be  user 
f  riendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1,  2.  Are  the  automated  and  automatic  requirements  as  specified 
in  Data  Dictionary  Processes  2. 1.5. 2  -  2. 1.5. 3  met? 

CI-1 

3.  Are  the  manual  requirements  as  specified  in  Data  Dictionary 
Process  2. 1.5.2  met? 

CI-1 

4a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

4b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 
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LEVEL  OF  T&E  REQUIRED: 

1,  2.  T  see  Software  Test  Plan 
3.  I  expert  judgment 

4a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
4b.  A  user  surveys 
SCHEDULED  TEST  DATE: 

1,  2.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

3.  Review  completed  within  10  days  following  development;  see 
schedule,  Appendix  G. 

4a.  Part  of  Alpha  Test  procedures. 

4b.  Surveys  administered  during  Beta  Tests  for  1,  2,  3. 
RESOURCES/FACILITIES  REQUIREMENTS : 

1,  2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 

and/or  real  data. 

3.  DAC/AF  experts  in  Buildings  1808  and  428. 

4a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

4b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1,  2.  Software  Alpha  Test  procedures. 

3.  Inspector  Review  Form. 

4a.  Software  Problem  Report. 

4b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 


N/A 


TEST  DATA  COLLECTION  REQUIREMENTS: 

1,  2.  See  Software  Test  Plan  for  further  details. 

3.  DAC/AF  expert (s)  review  of  off-line  plans  and  products. 

4a.  See  Software  Test  Plan  for  further  details. 

4b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1,  2.  Analysis  of  Alpha  Test  results. 

3.  Descriptive  judgments. 

4a.  AnaJysis  of  Alpha  Test  results. 

4b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1 ,  2 .  Go/No  Go  Status . 

3.  Expert  judgment  that  product (s)  satisfies  requirements. 

4a.  Go/No  Go  Status. 

4b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 


TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.2 _ 

TEST  ELEMENT :  Performance  Evaluation  Component _ 

TEST  OBJECTIVE (S) : 

1.  The  component  shall  provide  the  capability  for  the  user  to  ad¬ 
minister  on-  or  off-line  requisite  knowledge  tests  and  off-line 
performance  evaluations,  and  collect  knowledge  performance  and 
task  performance  evaluation  results. 

2.  The  component  shall  provide  the  capability  for  the  user  to 
permit  an  airman  to  bypass  unnecessary  training  requirements  based 
on  pretest  results. 

3.  The  component  shall  be  compatible  and  interactive  within  its 
own  subcomponents. 

4.  The  component  shall  have  the  capability  to  interface  with  new 
evaluation  strategies  that  may  become  available  during  the  con¬ 
tract  period. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

Cl— 4 

1.  Does  the  Performance  Evaluation  Component  provide  the  user 
with  acceptable  methods  and  tools  needed  to  administer  evaluation 
instruments,  and  collect  knowledge/performance  evaluation  results? 

CI-l 

2.  Does  the  Performance  Evaluation  Component  provide  the  user 
with  the  capability  to  "override”  standard  training  requirements 
based  on  pretest  results? 

CI-l 

3.  Do  the  subcomponents  interact  effectively? 


CI-l  .  . 

4.  Can  the  system  accommodate  new  Evaluation  strategies? 

LEVEL  OF  T&E  REQUIRED: 

1.  E  user  acceptance  survey  results  -  component  level 

2.  A  descriptive  analysis  of  summary  data  on  expected  outcomes 

3.  T  see  Software  Test  Plan  for  details  on  interactive  tests 

4.  I  expert  judgment 
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SCHEDULED  TEST  DATE: 


1.  Surveys  administered  during  Beta  Tests  for  1,  2,  3. 

2,  3.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

4.  Review  completed  within  10  days  following  development;  see 
schedule,  Appendix  G. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1.  AF  Workstation/AF  Workgroup. 

2,  3.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

4.  DAC/AF  experts  in  Buildings  1808  and  428. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  User  surveys  (see  sample  for  software  products) . 

2,  3.  Software  Alpha  Test  procedures. 

4.  Inspector  Review  Form. 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  AF  reactions  to  use  of  products. 

2,  3.  See  Software  Test  Plan  for  further  details. 

4.  DAC/AF  expert (s)  review  of  off-line  plans  and  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Descriptive  individual  item  and  composite  score  data  analysis. 

2,  3.  Analysis  of  Alpha  Test  results. 

4.  Descriptive  judgments. 
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TEST  CRITERIA  GOALS: 


1.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 

2,  3.  Go/No  Go  Status. 

4.  Expert  judgment  that  product (s)  satisfies  requirements. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.2 .1 

TEST  ELEMENT :  Task  Performance  Evaluation 


TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  administer  task  requisite  knowledge  test  on-line  and 
task  requisite  knowledge  tests  and  task  performance  evaluations 
off-line. 

2.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 
2.2.1  met? 

CI-1 

2a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

2b.  Do  the  users  of  the  plans,  tools  and  procedures,  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
2b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

2a.  Part  of  Alpha  Test  procedures. 

2b.  Surveys  administered  during  Beta  Tests  for  1. 
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RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/ dummy 
and/or  real  data. 

2a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

2b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2a.  Software  Problem  Report. 

2b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS'; 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2a.  See  Software  Test  Plan  for  further  details. 

2b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2a.  Analysis  of  Alpha  Test  results. 

2b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2a.  Go/No  Go  Status. 

2b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  ox*  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ _ 

B1  SPEC  REQUIREMENT  REFERENCE  NO . :  2.1 .2.2 _ 

TEST  ELEMENT :  Requisite  Knowledge  Test  Administration _ 

TEST  OBJECTIVE (3) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  administer  and  score  task  requisite  knowledge  tests 
on-line,  and  report  test  results  to  other  subcomponents  as 
required. 

2.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  print  requested  off-line  task  requisite  knowledge 
tests  and  performance  evaluation  materials,  and  to  accept  and 
score  off-line  test  results,  and  report  test  results  to  other  sub¬ 
components  as  required. 

3.  The  subcomponent  plans,  tools,  and  procedures  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1,  2.  Are  the  requirements  as  specified  in  Data  Dictionary 

Process  2.2.2  met? 

CI-1 

3a.  Does  the  associated  hardware  support  the  Software  products? 
CI~4 

3b.  do  the  users  of  the  plans,  tools,  and  procedures  rate  each 
as  satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

i,  2.  T  see  Software  Test  Plan 

3a.  I  Software  Problem  Report  results,  see  Software  Test  Plan 
3b,  A  user  survey 
SCHEDULED  TEST  DATE: 

ir  2.  Alpha  Test  immediately  following  software  development;  see 
schedule.  Appendix  G. 

3a.  Part  of  Alpha  Test  procedures. 

3b.  Surveys  administered  during  Beta  Tests  for  1,  2. 
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RESOURCES/FACILITIES  REQUIREMENTS : 

1,  2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 

and/or  real  data. 

3a?  DAC/AF  inspectors  for  review  of  Software  Problem  Reporbs. 

3b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1,  2.  Software  Alpha  Test  procedures. 

3a.  Software  Problem  Report. 

3b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1,  2.  See  Software  Test  Plan  for  further  details. 

3a.  See  Software  Test  Plan  for  further  details. 

3b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1/  2.  Analysis  of  Alpha  Test  results. 

3a.  Analysis  of  Alpha  Test  results. 

3b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS : 

1 ,  2 .  Go/No  Go  Status . 

3a.  Go/No  Go  Status. 

3b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 


A— 55 


TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

B-l  SPEC  REQUIREMENT  REFERENCE  NO.:  3. 7. 2. 3 
TEST  ELEMENT :  Performance  Observation _ 


TEST  OBJECTIVE (S ) : 

1.  Off-line.  The  subcomponent  shall  provide  guidelines  and 
standardized  administration  and  scoring  procedures  for  the  user  to 
use  when  administering  performance  evaluation. 

2.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  system  to  accept  performance  evaluation  pass/fail  results  and 
report  the  results  to  other  components  as  reguired. 

3.  The  subcomponent  plans,  tools,  and  procedures  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  manual  requirements  as  specified  in  Data  Dictionary 
Process  2.2.3  met? 

CI-l 

2.  Are  the  automated  requirements  as  specified  in  Data  Dictionary 
Process  2.2.3  met? 

CI-1 

3a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

3b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  I  expert  judgment 

2.  T  see  Software  Test  Plan 

3a.  I  Software  Problem  Report  results,  see  Software  Test  Plan 
3b.  a  user  survey 
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SCHEDULED  TEST  DATE: 


1.  Review  completed  within  10  days  following  development;  see 
schedule,  Appendix  G. 

2.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

3a.  Part  of  Alpha  Test  procedures. 

3b.  Surveys  administered  during  Beta  Tests  for  1,  2,  3. 
RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC/AF  experts  in  Buildings  1808  and  428. 

2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

3a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

3b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Inspector  Review  Form. 

2.  Software  Alpha  Test  procedures. 

3a.  Software  Problem  Report. 

3b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  DAC/AF  expert(s)  review  of  off-line  plans  and  products. 

2.  See  Software  Test  Plan  for  further  details. 

3a.  See  Software  Test  Plan  for  further  details. 

3b.  AF  reactions  to  use  of  products. 
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DATA  ANALYSIS  REQUIREMENTS: 

1.  Expert  judgment  that  product (s)  satisfies  requirements. 

2.  Analysis  of  Alpha  Test  results. 

3a.  Analysis  of  Alpha  Test  results. 

3b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Expert  judgment  that  product (s)  satisfies  requirements. 

2.  Go/No  Go  Status. 

3a.  Go/No  Go  Status. 

3b.  Majority  of  users  rate  product  satisfactory?  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION 


Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3. 7. 2. 4 

TEST  ELEMENT:  Access  Control _ 


TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  control  access  to  evaluation  materials  on-line  by 
granting  individuals  access  to  various  Editors  and  by  granting 
various  permissions  to  functions  within  each  Editor?  and  to  con¬ 
trol  the  receipt,  storage,  and  delivery  of  all  evaluation 
materials  during  on-line  test  presentations. 

2.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  control  access  to  printed  evaluation  materials  (off¬ 
line)  by  allowing  the  user  to  enter  a  delivery  and  return  notice 
containing  name,  SSAN,  date,  and  time  of  delivery/return,  and  a 
materials  destroyed  notice  when  applicable. 

NOTE:  Control  of  the  actual  printed  evaluation  materials  is  a 

manual  process  to  be  developed  by  the  Air  Force  and,  therefore, 
not  subject  to  evaluation  under  the  MTP. 

3.  Off-line.  The  subcomponent  shall  provide  the  user  with 
guidelines  for  off-line  procedures  for  storage,  distribution,  and 
collection  of  evaluation  materials  and  devices  for  job  site  task 
performance  evaluations. 

4.  The  subcomponent  plans,  tools,  and  procedures  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1,  2.  Are  the  automated  and  automatic  requirements  as  specified 
in  Data  Dictionary  Processes  2. 1.5. 2  -  2. 1.5. 3  met? 

CI-l 

3.  Are  the  manual  requirements  as  specified  in  Data  Dictionary 
Process  2. 1.5. 2  met? 

CI-1 

4a.  Does  the  associated  hardware  support  the  software  products? 


CI-4 

4b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1/2.  T  see  Software  Test  Plan 
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3.  I  expert:  judgment 

4a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
4b.  A  user  surveys 
SCHEDULED  TEST  DATE: 

1,  2.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

3.  Review  completed  within  10  days  following  development;  see 
schedule.  Appendix  G. 

4a.  Part  of  Alpha  Test  procedures. 

4b.  Surveys  administered  during  Beta  Tests  for  1,  2,  3. 
RESOURCES/FACILITIES  REQUIREMENTS : 

1,  2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 

and/or  real  data. 

3.  DAC/AF  experts  in  Buildings  1808  and  428. 

4a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

4b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1,  2.  Software  Alpha  Test  procedures. 

3.  Inspector  Review  Form. 

4a.  Software  Problem  Report. 

4b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 
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TEST  DATA  COLLECTION  REQUIREMENTS: 

1,  2.  See  Software  Test  Plan  for  further  details. 

3.  DAC/AF  expert (s)  review  of  off-line  plans  and  products. 

4a.  See  Software  Test  Plan  for  further  details. 

4b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1,  2.  Analysis  of  Alpha  Test  results. 

3.  Descriptive  judgments. 

4a.  Analysis  of  Alpha  Test  results. 

4b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1,  2.  Go/No  Go  Status. 

3.  Expert  judgment  that  product(s)  satisfies  requirements. 

4a.  Go/No  Go  Status. 

4b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3. 7. 2. 5 _ 

TEST  ELEMENT :  Accept  Off-line  Completion  Results _ 

TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  input  the  results  of  off-line  knowledge  tests,  perfor¬ 
mance  evaluations,  and/or  evaluation  cancellation  notices  into  the 
system. 

2.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 
2.2.5  met? 

CI-1 

2a.  Does  the  associated  hardware  support  the  software  products? 


Cl -4 

2b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
2b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

2a.  Part  of  Alpha  Test  procedures. 

2b.  Surveys  administered  during  Beta  Tests  for  1. 


! 

RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

2a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

2b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

,  1.  Software  Alpha  Test  procedures. 

2a.  Software  Problem  Report. 

2b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2a.  See  Software  Test  Plan  for  further  details. 

2b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2a.  Analysis  of  Alpha  Test  results. 

2b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2a.  Go/No  Go  Status. 

2b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

B-j^  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.2. 6 _ 

TEST  ELEMENT:  Collect  Performance  Evaluation  Data _ 

TEST  OBJECTIVE (S ) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  transfer  all  evaluation  results  to  a  data  store  where 
it  can  be  accessed  by  other  subcomponents  as  required. 

2.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 
2.2.6  met? 

CI-l 

2a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

2b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
2b.  A.  User  survey 
SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

2a.  Part  of  Alpha  Test  procedures. 

2b.  Surveys  administered  during  Beta  Tests  for  1,  2,  3. 
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RESOURCES/FACILITIES  REQUIREMENTS: 

1.  DAC  staff  utilizing  Air  Force  f ac i lit ies/ equipment/ dummy 
and/or  real  data. 

2a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

2b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2a.  Software  Problem  Report. 

2b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2a.  See  Software  Test  Plan  for  further  details. 

2b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2a.  Analysis  of  Alpha  Test  results. 

2b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2a.  Go/No  Go  Status. 

2b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 


TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

B1  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.3 _ 

TEST  ELEMENT :  Apply  Training  Quality  Control  Component _ 

TEST  OBJECTIVE (S) : 

1.  The  component  shall  provide  valid  and  reliable,  systematic 
task  performance  evaluations  to  assess  effectiveness  of  AOTS 
training. 

2.  The  component  shall  be  compatible  and  interactive  within  its 
own  subcomponents. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-4 

1.  Does  the  Apply  Training  Quality  Control  Component  provide  the 
user  with  acceptable  methods  and  tools  needed  to  assess  the  effec¬ 
tiveness  of  AOTS  training? 

CI-1 

2.  Do  the  subcomponents  interact  effectively? 

LEVEL  OF  T&E  REQUIRED: 

1.  E  User  Acceptance  Survey  Results,  component  level 

2.  T  see  Software  Test  Plan  for  details  on  integration  tests. 
SCHEDULED  TEST  DATE: 

1.  Surveys  administered  during  Beta  Test. 

2.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1.  AF  Workstation/ AF  Workgroup. 

2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  User  surveys  (see  sample  for  software  products) . 

2 .  Software  Alpha  Test  procedures . 
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BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  AF  reactions  to  use  of  products. 

2.  See  Software  Test  Plan  for  further  details. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Descriptive  individual  item  and  composite  score  data  analysis. 

2.  Analysis  of  Alpha  Test  results. 

TEST  CRITERIA  GOALS: 

1.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 


2 .  Go/No  Go  Status . 


TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

Bjl  SPEC  REQUIREMENT  REFERENCE  NO.:  3. 7. 3,1 _ 

TEST  ELEMENT:  Quality  Control  Evaluation  Event  Selection _ _ 

TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  activate  automatic  sampling  algorithms  to  select  an 
appropriate  task,  airman  candidate,  external  evaluator,  and  alter¬ 
nate  evaluator  for  a  training  quality  control  evaluation  event; 
and  send  a  request  for  the  selected  event  to  take  place.  Note: 
Establishment  of  selection  criteria  guidelines  will  be  the  respon¬ 
sibility  of  the  user  and  not  subject  to  evaluation  under  the 
Master  Test  Plan. 

2.  The  subcomponent  plans,  tools,  and  procedures  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Processes 
2.3. 1.1  -  2. 3. 1.4  met? 


CI-1 

2a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

2b.  Do  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
2b.  A  user  survey 
SCHEDULED  TEST  DATE: 

r.  Alpha  rest  immediately  following  software  development?  see 
schedule.  Appendix  G. 

2a.  Part  of  Alpha  Test  procedures. 

2b.  Surveys  administered  during  Beta  Tests  for  1. 
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RESOURCES/FACILITIES  REQUIREMENTS : 

1-  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

2a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

2b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2a.  Software  Problem  Report. 

2b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2a.  See  Software  Test  Plan  for  further  details. 

2b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2a.  Analysis  of  Alpha  Test  results. 

2b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2a.  Go/No  Go  Status. 

2b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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TEST  PLAN 


\ 


SUBSYSTEM:  EVALUATION 


Bx  SPEC  REQUIREMENT  REFERENCE  NO . :  3. 7. 3. 2 _ 

TEST  ELEMENT :  Quality  Control  Evaluation  Follow-up _ 

TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  automatically  receive  Q.C.  Event 
confirmat ion/cancellation  data  for  review  and  action. 

2.  On-line.  Following  manual  notification  of  Q.C.  results  and 
any  decertification  recommendations  to  training  manager,  unit  su¬ 
pervisor,  and  unit  commander  for  review  and  action,  the  subcom¬ 
ponent  shall  provide  the  capability  for  the  user  to  send  Q.C. 
results  follow-up  data  to  the  Airman  Training  Management  component 
for  review  and  further  action. 

3.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  enter  all  Q.C.  evaluation  events,  cancellations,  and 
Q.C.  results  follow-up  data  into  a  data  store  for  report  gener¬ 
ation. 

4.  The  subcomponent  plans,  tools,  and  procedures  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1/  2,  3.  Are  the  requirements  as  specified  in  Data  Dictionary 

Process  2.3.2  met? 

CI-1 

4a.  Does  the  associated  hardware  support  the  software  products? 


CI-4 

4b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1,  2,  3.  T  see  Software  Test  Plan 

4a.  I  Software  Problem  Report  results  —  see  Software  Test  Plan 
4b.  A  user  survey 
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SCHEDULED  TEST  DATE: 


1,  2,  3.  Alpha  Test  immediately  following  software  development; 
see  schedule.  Appendix  G. 

4a.  Part  of  Alpha  Test  procedures. 

4b,  Surveys  administered  during  Beta  Tests  for  1,  2,  3. 
RESOURCES/FACILITIES  REQUIREMENTS : 

1,  2,  3.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

4a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

4b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1,  2,  3.  Software  Alpha  Test  procedures. 

4a.  Software  Problem  Report. 

4b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1/  2,  3.  See  Software  Test  Plan  for  further  details. 

4a.  See  Software  Test  Plan  for  further  details. 

4b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1,  2,  3.  Analysis  of  Alpha  Test  results. 

4a.  Analysis  of  Alpha  Test  results. 

4b.  Descriptive  individual  item  and  composite  score  data 
analysis. 
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TEST  CRITERIA  GOALS: 


1,  2,  3.  Go/No  Go  Status. 

4a.  Go/No  Go  Status. 

4b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO . :  3. 7. 3. 3 _ 

TEST  ELEMENT :  Quality  Control  Report  Generation _ 

TEST  OBJECTIVE (S) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  generate  periodic  summary  reports  of  training  quality 
control  activities. 

2.  The  subcomponent  plans,  tools,  and  procedures  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 
2.3.3  met? 

CI-1 

2a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

2b.  Do  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
2b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule.  Appendix  G. 

2a.  Part  of  Alpha  Test  procedures. 

2b.  Surveys  administered  during  Beta  Tests  fcr  1,  2,  3. 
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RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

2a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

2b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2a.  Software  Problem  Report. 

2b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2a.  See  Software  Test  Plan  for  further  details. 

2b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2a.  Analysis  of  Alpha  Test  results. 

2b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2a.  Go/No  Go  Status. 

2b.  .  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ 

B1  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.4 _ 

TEST  ELEMENT :  System  Evaluation  Component _ 

TEST  OBJECTIVE (S) : 

1.  The  component  shall  provide  the  capability  for  the  user  to 
collect,  maintain,  and  report  data  regarding  the  performance  of 
the  prototype  AOTS  in  meeting  system  goals  for  training  quality 
and  task  performance. 

2.  The  component  shall  be  compatible  and  interactive  within  its 
own  subcomponents. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CT-4 

1.  Does  the  System  Evaluation  Component  provide  the  user  with  ac¬ 
ceptable  tools  and  methods  needed  to  generate  required  system 
level  reports? 

CI-1 

2.  Do  the  subcomponent6s  interact  effectively? 

LEVEL  OF  T&E  REQUIRED: 

1.  E  user  acceptance  survey  results  -  component  level 

2.  T  see  Software  Test  Plan  for  details  on  interactive  testing 
SCHEDULED  TEST  DATE: 

1.  Surveys  administered  during  Beta  Test. 

2.  Alpha  Test  immediately  following  software  development;  see 
schedule.  Appendix  G. 

RESOURCES/FACILITIES  REQUIREMENTS : 

1.  AF  Workstation/AF  Workgroup. 

2.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  User  surveys  (see  sample  for  software  products) . 

2.  Software  Alpha  Test  procedures. 
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BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  AF  reactions  to  use  of  products. 

2.  See  Software  Test  Plan  for  further  details. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Descriptive  individual  item  and  composite  score  data  analysis. 

2.  Analysis  of  Alpha  Test  results. 

TEST  CRITERIA  GOALS: 

1.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 


2.  Go/No  Go  Status. 


TEST  PLAN 


SUBSYSTEM:  EVALUATION 


Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3.7.4. 1 _ 

TEST  ELEMENT:  Unit  Training  Report  Generation _ 

TEST  OBJECTIVE (S ) : 

1.  On-line.  The  subcomponent  shall  provide  the  capability  for 

the  user  to  collect,  maintain,  and  report  up-to-date  to  system 
managers  in  reports  that  summarize  unit  training  accomplished, 
training  goal  status,  and  training  and  performance  trends. 
Reports  generated  shall  include  standard  reports  generated  at  pe¬ 
riodic  intervals  for  specified  recipients,  and  ad  hoc  reports  gen¬ 
erated  on  demand  to  address  specific  AOTS  performance  aspects. 
Note:  Requests  for  data  verification  and  user  defined  ad  hoc 

reports  are  a  manual  process  and  not  subject  to  evaluation  imder 
the  Master  Test  Plan. 

2.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  requirements  as  specified  in  Data  Dictionary  Processes 
2.4. 1.1  -  2.4. 1.4  met? 


CI-1 

2a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

2b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 

LEVEL  OF  T&E  REQUIRED: 

1.  T  see  Software  Test  Plan 

2a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
2b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

2a.  Part  of  Alpha  Test  procedures. 

2b.  Surveys  administered  during  Beta  Tests  for  1. 
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RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

2a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

2b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Software  Alpha  Test  procedures. 

2a.  Software  Problem  Report. 

2b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 

TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  See  Software  Test  Plan  for  further  details. 

2a.  See  Software  Test  Plan  for  further  details. 

2b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Analysis  of  Alpha  Test  results. 

2a.  Analysis  of  Alpha  Test  results. 

2b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Go/No  Go  Status. 

2. a.  Go/No  Go  Status. 

2b.  _  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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TEST  PLAN 


SUBSYSTEM:  EVALUATION _ _ _ 

Bx  SPEC  REQUIREMENT  REFERENCE  NO.:  3. 7. 4. 2 _ 

TEST  ELEMENT:  System  Effectiveness  Report  Generation _ 

TEST  OBJECTIVE (S) : 

1.  Off-line.  The  subcomponent  shall  provide  the  guidelines  and 
mechanisms  for  the  user  to  manually  collect  evaluation  materials 
development  data  generated  during  the  development  process. 

2.  Off-line.  The  subcomponent  shall  provide  the  capability  for 
the  user  to  store  evaluation  materials  development  data  for  use 
in  the  generation  of  standard  and  ad  hoc  system  effectiveness 
reports . 

3.  On-line.  The  subcomponent  shall  provide  the  capability  to 
collect,  maintain,  and  report  up-to-date  information  to  system 
managers.  The  reports  shall  provide  an  analysis  of  system  data 
reflecting  the  effectiveness  of  the  AOTS.  Reports  generated  shall 
include  standard  reports  generated  at  periodic  intervals  for 
specified  recipients,  and  ad  hoc  reports  generated  on  demand  to 
address  specific  AOTS  performance  aspects.  Note:  Requests  for 
data  verification  and  user  defined  ad  hoc  reports  are  a  manual 
process  and  not  subject  to  evaluation  under  the  Master  Test  Plan. 

4.  The  subcomponent  plans,  procedures,  and  tools  shall  be  user 
friendly  and  supportable. 

EVALUATION  QUESTION (S)  TO  BE  ANSWERED: 

CI-1 

1.  Are  the  manual  requirements  as  specified  in  Data  Dictionary 
Process  2. 4. 2. 5  met? 

CI-1 

2.  Are  the  requirements  as  specified  in  Data  Dictionary  Process 
2 . 4 . 2 . 5  met? 

CI-1 

3.  Are  the  requirements  as  specified  in  Data  Dictionary  Processes 
2. 4. 2.1  -  2. 4. 2. 4  met? 

CI-1 

4a.  Does  the  associated  hardware  support  the  software  products? 
CI-4 

4b.  Do  the  users  of  the  plans,  tools,  and  procedures  rate  each  as 
satisfactory? 
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LEVEL  OF  T&E  REQUIRED: 

1.  I  expert  judgment 

2,3.  T  see  Software  Test  Plan 

4a.  I  Software  Problem  Report  results  -  see  Software  Test  Plan 
4b.  A  user  survey 
SCHEDULED  TEST  DATE: 

1.  Review  completed  within  10  days  following  development;  see 
schedule,  Appendix  G. 

2,  3.  Alpha  Test  immediately  following  software  development;  see 
schedule,  Appendix  G. 

4a.  Part  of  Alpha  Test  procedures. 

4b.  Surveys  administered  during  Beta  Tests  for  1,  2,  3. 
RESOURCES/FACILITIES  REQUIREMENTS : 

1.  DAC/AF  experts  in  Buildings  1808  and  428. 

2,  3.  DAC  staff  utilizing  Air  Force  facilities/equipment/dummy 
and/or  real  data. 

4a.  DAC/AF  inspectors  for  review  of  Software  Problem  Reports. 

4b.  AF  Workstation/AF  Workgroup. 

EVALUATION  INSTRUMENTS  REQUIREMENTS: 

1.  Inspector  Review  Form. 

2,  3.  Software  Alpha  Test  procedures. 

4a.  Software  Problem  Report. 

4b.  User  surveys  (see  sample  for  software  products) . 

BASELINE  DATA  COLLECTION  REQUIREMENTS: 

N/A 
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TEST  DATA  COLLECTION  REQUIREMENTS: 

1.  DAC/AF  expert (s)  review  of  off-line  plans  and  products. 

2,  3.  See  Software  Test  Plan  for  further  details. 

4a.  See  Software  Test  Plan  for  further  details. 

4b.  AF  reactions  to  use  of  products. 

DATA  ANALYSIS  REQUIREMENTS: 

1.  Descriptive  judgments. 

2,  3.  Analysis  of  Alpha  Test  results. 

4a.  Analysis  of  Alpha  Test  results. 

4b.  Descriptive  individual  item  and  composite  score  data 
analysis. 

TEST  CRITERIA  GOALS: 

1.  Expert  judgment  that  product(s)  satisfies  requirements. 

2,  3.  Go/No  Go  Status. 

4a.  Go/No  Go  Status. 

4b.  Majority  of  users  rate  product  satisfactory;  review  of 
specific  comments  for  possible  revisions  or  additional  training. 
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INTRODUCTION 


In  accordance  with  this  Master  Test  Plan,  AOTS  products  un¬ 
dergo  alpha  testing  (informal  and  formal  software  testing)  and 
beta  testing  (initial  use  and  evaluation  by  users)  prior  to  be¬ 
coming  operational. 

Alpha  testing  will  ensure  that  the  software  meets  the  func¬ 
tional  requirements,  operates  correctly,  and  is  satisfactorily 
supported  by  the  hardware.  Alpha  testing  will  be  governed  by  Ap¬ 
pendix  D,  the  Software  Test  Plan  (STP) ,  and  will  be  conducted  by 
DAC  software  development  personnel  using  real  and  simulated  data 
’oases  as  appropriate.  The  Software  Problem  Report  (pp.  B-14, 
-15)  will  be  used  to  document  difficulties,  and  will  be  the  basis 
for  initiating  corrective  actions.  Sample  procedural  checklists 
for  software  follow  (pp.  B-ll  to  B-13) . 

Beta  testing  of  software  products  will  begin  after  alpha 
testing  is  completed.  Beta  testing  will  be  in  accordance  with  the 
Master  Test  Plan,  and  will  be  conducted  by  DAC  with  both  DAC  and 
AF  personnel  using  and  evaluating  the  software  products.  Problem 
Reports  (p.  B-16)  will  be  used  to  document  difficulties  and  will 
be  the  basis  for  corrective  actions.  During  beta  testing, 
products  will  be  evaluated  for  user  friendliness  by  means  of  user 
surveys  (see  pp.  B-4,  to  B-7)  .  Experts  (DAC  and  AF)  will  use 
procedure  checklists  (sample  on  pp.  B-10,  -11)  and  inspection 
review  forms  (see  pp.  B-8,  -9)  to  establish  that  AOTS  products 
meet  requirements. 

Whenever  "expert  judgment"  or  "panel  of  experts"  is  called 
for  during  testing,  the  DAC  Test  Manager  will  designate  one  or 
more  DAC  personnel  with  knowledge  of  the  product  to  serve  as  the 
expert (s)  for  testing.  The  AF  Program  Manager  or  designee  may, 
as  desired,  designate  one  or  more  AF  personnel  to  serve  in  this 
expert  capacity  for  any  of  the  testing  activities.  The  Test  Work¬ 
ing  Group  will  serve  as  clearing  house  for  discussion  and  recom¬ 
mendations  as  to  DAC  and  AF  experts. 
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PROCEDURAL  GUIDE  FOR  USER  SURVEY 


1.  Users  of  the  product  (during  Phase  II,  usually  DAC  and  AF 
personnel)  will  be  given  time  to  become  acquainted  with  the 
product. 

2.  Users  will  then  respond  to  a  survey  where  they  will  indicate 
their  level  of  agreement  to  each  item  of  concern  on  a  7-point 
opinion  scale.  The  surveys  for  software  products  will  contain 
standard  statements  to  aid  in  cross  comparisons  of  products. 
When  required,  the  survey  for  a  particular  product  will  contain 
additional  items  of  specific  concern.  The  users  will  be  provided 
directions  and  space  for  recording  specific  reactions,  problems, 
or  questions  related  to  the  product.  These  will  be  tabulated  and 
analyzed  separately  from  the  descriptive  item  analysis. 

Each  survey  will  contain  general  directions  to  the  survey  ad¬ 
ministrator  for  setting  a  positive  climate  for  administering  the 
survey.  Standard  directions  to  the  users  will  also  be  contained 
on  the  survey  instrument. 

3.  Results  will  be  tabulated  by  frequency  and  percent  on  an  item 
by  item  basis,  and  across  items  for  a  total  score.  A  product 
would  satisfy  the  user  friendliness  goal  by  reaching  the  follow¬ 
ing  test  criteria  goal. 

Product  is  rated  satisfactory  or  higher  on  each 
item  by  70%  of  those  surveyed. 

Alternate  goals  (could  be  considered) : 

Product  is  rated  satisfactory  or  higher  on  all 
critical  items  by  80%  of  those  surveyed,  and  on  all 
other  items  by  50%  of  those  surveyed. 

Each  item  response  is  given  a  relative  value, 
with  Strongly  Agree  (SA)  coded  7  and  Strongly 
Disagree  (SD)  coded  1.  An  average  score  for  each 
item  is  calculated,  and  each  item  must  meet  or  exceed 
minimum  acceptance  value  (4.0)  or  be  flagged  for 
review. 
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sample;  aots  user  survey 


TEST  EDITOR  SURVEY 


DIRECTIONS :  Please  complete  this  survey  without  conferring  with 

other  1ST  members.  The  opinions  you  give  should  be  your  own. 

Enter  your  name,  SSAN,  and  the  date  in  the  appropriate  spaces. 
Indicate  your  response  to  each  statement  by  circling  the  number 
that  corresponds  with  your  opinion  of  the  item.  Specific  reac¬ 
tions,  comments,  and  explanation  may  be  added  at  the  end  of  the 
survey. 

NAME _ SSAN _ 

DATE 


1.  I  found  it  easy  to  learn  to  use  the  Test  Editor. 

1  2  3  4  5  6  7 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 


2.  The  Test  Editor  was  predictable  and  consistent. 


1 

2 

3 

4 

5 

6 

7 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

3.  I  felt  I  controlled  the  system  as  much  as  I  needed. 

1  2  3  4  5  6  7 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 


4.  I  could  leave  the  Test  Editor  without  losing  data. 

1  2  3  4  5  6  7 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 
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5.  The  editing  capability  (being  able  to  correct  or  change 
entries)  was  easy  to  use. 

1  2  3  4  5  6  7 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 


6.  The  displays  were  uncluttered. 

1  2  3  4  5  6  7 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 


7.  The  displays  were  pleasing  to  the  eye. 

1  2  3  4  5  6  7 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 


8.  The  displays  gave  me  enough  information  to  do  my  work. 

1  2  3  4  5  6  7 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 


9.  The  displays  had  a  consistent  layout. 

1  2  3  4  5  6  7 


STRONGLY  NEUTRAL 

DISAGREE 


STRONGLY 

AGREE 


I  did.  not  require  a  computer  background  to  understand  the 
information  displayed  on  the  screen. 


2  3  4  5  6  7 


STRONGLY  NEUTRAL 

DISAGREE 


STRONGLY 

AGREE 


I 
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11.  The  prompts  (messages  at  the  bottom  of  the  screen)  were 
clear  and  understandable. 


STRONGLY 

DISAGREE 


NEUTRAL 


STRONGLY 

AGREE 


12.  The  commands — for  example,  (E)XIT,  (N)EXT  PAGE — did  what  I 
expected. 


STRONGLY 

DISAGREE 


NEUTRAL 


STRONGLY 

AGREE 


13 .  The  error  messages  in  the  Test  Editor  were  descriptive  and 
helpful . 


STRONGLY 

DISAGREE 


NEUTRAL 


STRONGLY 

AGREE 


14.  The  Test  Editor  responded  quickly  to  my  commands. 


STRONGLY 

DISAGREE 


NEUTRAL 


STRONGLY 

AGREE 
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SOFTWARE  PROBLEM  REPORT  (SPR) 


FROM 


DATE  SUBMITTED 


INFO  COPIES  TO 


AOS 

PROGRAM  NAME 

IDENTIFICATION 

RUN  OATE 

POINT  OF  CONTACT  (Type  or  print  name) 

B-8a 


PROBLEM  REPORT 


Log  No. 


Name : 


Date: 


Time: 


Locati on : 


Telephone  number:  _ 


Hardware  being  used  when  problem  was  detected:  []  Z248  []  8600 
[]  Other:  _ 

Description  of  problem:  _ 


Recommendation  of  Test  Working  Group  | 


Date: 


Priority:  []  Immediate  []  Delayed  []  Workaround  []  Other: 

Recommended  Corrective  action: 


Date  work  completed: 


CM:  [  3  CM  Entered  []  CCB  approved  []  CM  Action: 

P  Bug  ^  User  Error  []  Class  1  (Enhancement) 
Corrective  action  approval: 


|  Software  Manager  Configuration  Mgt, 

AOTS  0100-24  (Rev. _ )  Nov.  17,  1986 


Program  Manager 
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APPENDIX  C 
MASTER  TEST  PLAN 
PROCESS  DATA 


C  -  1 


PROCESS  DATA  PROCEDURES 


Process  data  will  involve  the  acquisition  of  Deployment  Observa¬ 
tions,  hotline  records,  and  Log-On  data.  This  data  will.be  col¬ 
lected  during  the  AOTS  prototype  deployment  period  and  used  to 
provide  ongoing  feedback  on  the  utilization  of  AOTS.  Moreover, 
this  data  will  be  used  to  enhance  the  interpretation  of  the  out¬ 
come  measures  collected  during  and  near  the  end  of  the  prototype 
deployment  period. 

C.l  WORKCENTER  OBSERVATION  FORMS 

C.1.1  Data  to  be  Collected.  Workcenter  observation  data  in¬ 
volves  notations  and  comments  on  how  the  software  and  hardware 
are  functioning  at  the  workcenter  level.  In  addition  to  collect¬ 
ing  descriptive  information  (i.e.,  user  AFSC,  user  position,  and 
workcenter)  ,  computer  down  time  will  be  recorded;  and  the 
functioning  of  AOTS  software  tools  and  hardware  components  will 
be  observed  and  documented. 

C.l. 2  Requirements  for  DCRs.  The  DCRs  will  be  AOTS  1ST  and 
staff  members  who  will: 

1.  Learn  to  conduct  an  observation  during  visits  to  the 
workcenters,  using  the  Deployment  Observation  Form, 
Figure  C.l,  p.  7. 

2.  Accurately  document  observations  on  said  form. 

C.l. 3  Frequency  of  Data  Collection.  Behavioral  observations 
will  be  conducted  at  least  once  a  week  per  workcenter.  Documen¬ 
tation  of  the  observations  will  be  completed  during  each  visit 
to  the  workcenters. 

C.l. 4  Data  Collection  Procedure.  Members  of  the  1ST  will  act  as 
the  DCRs  for  the  workcenters  within  their  respective  AFSs. 

C.l. 4.1  Complete  Deployment  Observation  Form. 

1.  Enter  date  ana  identifying  information  at  the  top  of  the 
form  (i.e.,  observer,  user  AFSC,  workcenter,  user  level) 
Figure  C.2,  p.  8. 

2.  Record  a  notation  of  + (positive),  -(negative),  or  o (neu¬ 
tral)  to  the  left  of  the  software  function  and/or  hard¬ 
ware  component. 
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3 .  Record  a  brief  statement  in  the  comment  section  to 
explain  the  notation.  For  example,  if  you  observe  a 
trainer  effectively  using  the  ATR,  and  he  responds  fav¬ 
orably  when  you  ask  him  about  the  ATR,  you  would  enter 

+  A  10  (reference  code  for  the  ATR  function)  then  a  brief 
explanation  of  the  observation.  Similar  procedures  will 
apply  for  entries  pertaining  to  the  hardware. 

4 .  Conduct  observations  in  the  workcenters  during  computer 
down  times  resulting  from  problems  with  the  communica¬ 
tion  lines,  VAX  8650,  etc..  Document  the  duration  of 
down  time,  impact  on  OJT,  continued  OJT  activity  during 
this  down  period,  and  the  process  of  re-entering  the 
system  when  the  computer  is  functioning  again.  -Because 
the  down  time  period  may  be  prolonged,  a  follow-up 
visit  may  be  necessary  to  collect  this  information  from 
the  workcenters. 

C.1.5  Time  Required  for  Data  Collection.  Time  needed  to  conduct 
observations  at  the  workcenters  and  document  information  is  as 
follows : 


30  -  45  min.  Conduct  observation 
15  min.  Complete  observation  form 

C.1.6  Disposition  of  Data.  After  the  observation  forms  have 
been  completed,  they  will  be  delivered  to  the  1ST  Superinten¬ 
dent,  who  will  conduct  a  quality  control  check  of  the  data  to  as¬ 
sure  that  accurate  and  understandable  information  is  on  the  ob¬ 
servation  form.  The  quality  control  will  involve: 

-  Check  to  see  that  all  descriptive  information  has  been  ac¬ 
curately  entered  on  the  form; 

-  Check  to  assure  that  an  explanation  of  the  observation  has 
been  entered  in  the  comment  section. 

These  observation  forms  will  be  forwarded  to  T-l  for  analysis  by 
the  AF  staff.  Analysis  of  the  data  will  consist  of  descriptive 
statistics  and  content  analysis. 


C . 2  AOTS  HOTLINE  PROBLEM  REPORT 

c  2.1  Data  to  be  collected.  The  AOTS  Hotline  Problem  Report  is 
designed  to  document  the  incoming  calls  from  AOTS  users  needing 
assistance  with  various  problems  encountered  during  the  deploy¬ 
ment  period.  This  form  will  involve  recording  the  specific 
AOTS  function,  type  of  malfunction,  specific  information  regard- 
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ing  the  problem,  and  resolution  or  recommended  action. 

C.  2.2  Requirements  for  DCR.  DCR  will  be  composed  of  a  desig¬ 
nated  1ST  member  who  will: 

1.  Learn  to  respond  to  calls  made  on  the  AOTS  hotline, 

2.  Accurately  document  information  on  the  hotline  report 
form. 


C.2.3  Frequency  of  Data  Collection.  The  frequency  of  the  data 
collection  will  be  contingent  on  the  number  of  incoming  calls  on 
the  hotline. 


C . 2 . 4  Data  Collection  Procedure . 


C.2.4.1  Complete  AOTS  Hotline  Problem  Report. 

1.  On  the  upper  portion  of  the  form,  enter  the  caller's 
name  phone  number,  date  of  call,  time  of  call,  and 
workcenter. 

2.  In  terms  of  the  reported  problem,  enter  the  following 
information: 

-  specific  OJT/computer  function  that  is  being  at¬ 
tempted  (e.g.,  log-on,  ATR) , 

-  description  of  the  malfunction, 

-  question  being  asked  to  resolve  the  problem, 

-  individual  providing  the  resolution, 

-  action  taken/recommended  to  resolve  the  problem, 

-  comments  that  may  provide  additional  information. 


3.  On  the  bottom  of  the  form,  enter  the  person  recording 
the  problem  report. 


C.2.5.  Time  Required  for  Data  Collection.  Time  needed  to  com¬ 
plete  information  on  the  problem  report  should  be,  in  most  cases, 

t.r  ■?  4-  V*v  -i  V-v  '  C  wn  n  J  -  .........  4  ....  ...  .  .1.  4  ..  ... 

»» w  i  j-  *  *  mxuuuco  '-axis  ic^uiucu  uu  une  aiK=>  wtijL  xiiCj  mctCii  me 

should  be  returned  within  24  hours  of  the  recorded  message. 


C.2.6  Disposition  of  Data.  After  the  problem  report  is  com¬ 
pleted,  it  will  be  submitted  to  the  AOTS  Hotline  NCOIC.  He  will 
perform  a  quality  control  check  to  assure  complete  and  under¬ 
standable  entry  of  the  information.  After  the  quality  control 
check,  this  data  will  be  submitted  for  analysis  at  T-l.  AF  Tech¬ 
nical  Advisor  will  use  descriptive  analysis  to  analyze  the.  data. 
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C.3  LOG-ON  DATA 

C.3.1  Data  to  be  collected.  The  log-on  data  consists  of  a  com¬ 
puter  program  that  records  the  utilization  of  the  AOTS  products 
at  the  workcenter  level.  More  specifically,  this  program  iden¬ 
tifies  the  user  by  ID  number,  records  the  date,  entry/exit  time 
when  user  logs  on/off  the  system,  specific  AOTS  tools/functions 
that  have  been  used,  and  the  CPU  time  spent  on  each 
tool/ function. 

C.3. 2  Frequency  of  data  collection.  The  log-on  computer  program 
will  be  accessed  on  the  VAX  at  Brooks  AFB  on  a  bimonthly 
basis. 

C.3. 3  Data  Collection  Procedure.  The  log-on  computer  program 
will  record  the  utilization  of  the  AOTS  functions  on  a  daily 
basis.  Every  two  weeks,  this  program  will  be  converted  to  an  AS¬ 
CII  text  file  for  statistical  analysis.  This  ASCII  file  will  be 
accessed  on  the  Zenith  248  at  T-I  for  completion  of  the  analysis. 

The  following  is  the  type  of  data  that  will  be  available  for 
analysis : 

-  Log-on  procedures:  User  Log-On,  program  start,  program 
end; 

-  Type  of  users:  Trainee,  Trainer,  Supervisor,  Training 
Manager,  Evaluator; 

-  Log  records:  Activity  date,  Activity  time,  System  Start, 
User  ID  (SSAN)  ,  program  (function)  name,  CPU  time. 

This  program  will  be  sorted  by  AFSC,  component,  and  user  type. 

C.3.4  Disposition  of  Data.  Statistical  analysis  of  the  Log-on 
data  is  as  follows: 

1.  Descriptive  analysis  of  data  to  determine  range,  mean, 
and  distribution  of  CPU  time.  This  data  will,  in  turn, 
be  sorted  by  AFS ,  components  and  user  type. 

2.  Basic  inferential  analysis  (e.g.,  t-tests,  linear  re¬ 
gression)  will  be  conducted  to  determine  differences 
within  and/or  between  AFSs  and/or  components. 

3.  Trend  analysis  within  AFSs,  components,  and  users  to 
determine  utilization  or  under-utilization  of  the  AOTS 
functions  across  time. 
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4.  Develop  user  groupings  (e.g.,  low  and  high  users  of  AOTS 
AOTS  functions)  to  interpret  and/or  add  meaning  to  the 
outcome  measures  (e.g.,  Acceptance  Survey,  Opinion  Sur¬ 
vey)  . 

5.  The  SAS  program  will  be  used  by  the  AF  technical  advisor 
to  analyze  this  data. 
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AOTS  DEPLOYMENT  OBSERVATION  FORM 


OBSERVER  _  USER'S  AFSC 

DATE _ 

WORKCEHTER _  USER'S  POSITION _ 


j  please  ado  a  +  for 

POSITIVE  COMMENTS 

0  FOR  NEUTRAL  COMMENTS 

1 

FOR  NEGATIVE  COMMENTS  ! 

J 

1.  SOFTWARE 

a.  management 

1.  LOGON 

2.  MTL 

3.  IM 

4.  TPP 

5.  SCHEDULER 

6.  RCPORTS 

7.  GPTR 

8.  OPTR 

9.  OTR 

10.  ATR 

11.  ITR 

12.  OTHER 

B.  EVALUATION 

1  POE 

2.  TI8 

3.  TE 

4.  TEST  PRESENTATION 

5.  SCORING  PROGRAM 

6.  OTHER 

C.  TRAINING  DELIVERY 

1.  GrEd 

2.  ISS 

3.  CAI 

4.  IVD 

5.  OTHER 

COMMENTS: 

II.  HARDWARE 

A.  CPU 

E.  OMR 

8.  SCREEN  C.  KEYBOARD 

F.  COMM  CA9LES  /  LINES  C.  IVD  PLAYER 

D.  PRINTER 

H.  OTHER 

COMMENTS: 

ill.  COMPUTER  DOWN  TIME 

DATE  COMPUTER  WAS  DOWN.. . .  DURATION  (TO  NEAREST  1  /4  HOUR): 

1  HOW  DOES  COMPUTER  DOWN  TIME  AFFECT  OJT"’ 


2.  WHAT  PROCEDURES  DO  THE  WORKCENTERS  HAVE  FOR  ACCOMPLISHING  THEIR  TRAINING  REQUIREMENTS 
DURING  THE  00 WN  TIME? 


*  3.  DO  WORKCENTERS  EXPERIENCE  ANY  PR08LEMS  RE-ENTERING  THE  SYSTEM  WHEN  IT'S  UP  AGAIN? 


COMMENTS: 


Figure  C . 1 


—  Example. 


AOTS  DEPLOYMENT  OBSERVATION  FORM 

OBSERVER  USER 


WORKCENTER_/£  AMp/f) 


USER'S  AFSC. 
USCR'S  POSITION. 


PLEaSC  add  A  -4  FOR  POSITIVE  COMMENTS  0  FOR  NEUTRAL  COMMENTS  -  TOR  NEGATIVE  COMMENTS 


I.  software 

A.  MANAGEMENT 

1.  LOGON 
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3.  IM 

4.  TPP 


7.  GPTR 

8.  OPTR 

9.  OTR 
-f  1 0.  ATR 


B.  EVALUATION 

1.  BOE 

2.  TIB 

3.  TE 


C.  TRAINING  DELIVERY 

1.  GrEd 

2.  ISS 

3.  CAI 


4.  TEST  PRESENTATION  4.  IVD 


5.  SCHEDULER  11.ITR 

6.  REPORTS  12.  OTHER 


5.  SCORING  PROGRAM 

6.  OTHER 


5.  OTHER 


COMMENTS:  . 
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II.  HARDWARE 

A.  CPU 
E.  OMR 

COMMENTS: 
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H.  OTHER 
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IP.  COMPUTE*"  DOWN  TIME 
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SupXAMtl  tUape!  pi/w  t  Mvitur  Mtmtflj/  fylMi's  ATA. 

2.  WHAT  PROCEDURES  00  THE  WORKCENTERS  HAVE  FOR  ACCOMPLISHING  THEIR  TRAINING 
REQUIREMENTS  DURING  THE  OOWN  TIME? 
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PREFACE 


This  document  was  prepared  for  the  AOTS  software  development 
team.  It  ?s  not  a  CDRL  item,  however,  it  is  t.-f erred  to  by  the 
CPCI  Development  Specifications,  Software  Development  Plan,  and 
Master  Test  Plan.  It  is  intended  to  be  used  both  as  a  standalone 
document,  in  low  level  software  development  testing,  and  in 
conjunction  with  the  Master  Test  Plan,  in  higher  level 
requirement  verification  testing. 

This  document  describes  formal  test  plans  for  system  testing 
and  acceptance  testing.  Acceptance  testing  was  deleted  from  the 
DAC  requirements  list.  However,  since  this  document  is  used  as  a 
stand-alone,  a  decision  was  made  to  include  the  document  as  a 
whole  to  avoid  future  confusion.  Any  mention  in  this  document  to 
acceptance  testing  should  be  disregarded  and  not  considered  in 
conjunction  with  the  Master  Test  Plan.  Only  procedures  detailing 
system  testing  should  be  referenced  in  conjunction  with  the  MTP. 

This  document  is  based  on  the  provisions  set  forth  in  Data 
Item  Description  (DID)  DI-MCCR-80014 .  This  DID  is  applicable  to 
DOD-STD-2167 ,  which  is  not  an  AOTS  applicable  document.  Some 
deviation  is  taken  from  this  DID  in  Sections  3  and  4.  In  Section 
3,  the  terminology  Computer  Program  Component  (CPC)  was 
substituted  for  Computer  Software  Component  (CSC) .  CPC  is 
consistent  with  the  AOTS  CPCI  Development  Specifications,  written 
according  to  the  21  March  79  version  of  MIL-STD-483.  CSC  is 
terminology  adopted  at  a  later  point  in  time  and  is  used  with 
DOD-STD-2167  documentation.  In  Section  4,  this  document  divides 
formal  testing  into  system  and  acceptance  testing.  The  DID  does 
not  make  this  distinction. 

DIDs  Dl-MCCR-80015 ,  Software  Test  Description,  and  DI-MCCR- 
80016,  Software  Test  Procedure,  were  not  followed  structurely  in 
the  Formal  Test  Procedure  model,  see  Attachment  C.  The  Formal 
Test  Procedure  contains  the  same  information  as  these  two  DIDs 
describe,  but  in  a  format  that  makes  formal  test  procedures 
consistent  with  informal  test  procedures.  The  same  is  true  for 
DID  DI-MCCR-80017 ,  Software  Test  Report,  and  the  Formal  Test 
Report  model  presented  in  Attachment  D. 

The  identification  of  AOTS  formal  test  procedures  in  Section 
4  is  complete  to  the  point  of  the  current  state  of  the  phased 
software  development  effort.  As  detailed  design  proceeds  on  the 
next  set  of  software,  these  tables  will  be  updated  accordingly. 
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1. 


SCOPE 


1.1  Identification.  This  Software  Test  Plan  establishes  the 
plan  for  testing  the  Computer  Program  Configuration  Items  (CPCIs) 
in  the  Advanced  On-the-job  Training  System  (AOTS)  Computer 
Support  Subsystem,  Software  Component.  The  CPCIs  are  identified 
as  the  Management  CPCI  as  described  in  the  development 
specification  numbered  70S647411;  the  Evaluation  CPCI,  70S647413; 
and  the  System  Support  CPCI,  70S647414. 

1.2  Purpose.  The  purpose  of  the  AOTS  is  to  test  a  design 
concept  that  will  apply  automated  support  to  increase  the 
efficiency  and  effectiveness  of  the  carrent  Air  Force  On-the-Job 
Training  (OJT)  system.  The  Management  CPCI  will  provide  software 
support  for  the  AOTS  Management  Subsystem,  70S647100.  It  will 
provide  software  to  support  the  functions  of  identifying  and 
managing  training  requirements  for  an  Air  Force  Speciality  (AFS) , 
and  managing  the  airmen  undergoing  training  in  the  AFS. 

The  Evaluation  CPCI  will  provide  software  support  for  the 
AOTS  Evaluation  Subsystem,  70S647300.  It  will  provide  software 
to  support  the  functions  of  developing  and  maintaining  evaluation 
instrumentation,  evaluating  performance  of  tasks  by  airmen, 
performing  training  quality  control  functions,  and  evaluating  the 
effectiveness  of  the  AOTS  as  a  system. 

The  System  Support  CPCI  will  be  composed  of  the  services 
required  by  the  other  CPCIs  to  interface  with  the  Hardware 
Component,  70S647401.  This  CPCI  will  perform  operating  system 
functions,  terminal  communication  and  data  base  input/output,  and 
will  provide  security  functions  to  control  access  to  the  system. 

1.3  Introduction.  This  plan  describes  AOTS  Phase  II  software 
testing.  Software  testing  is  divided  into  two  phases:  informal 
and  formal  testing.  Both  of  the  testing  phases  are  divided  into 
two  levels.  Unit  and  integration  level  testing  are  done  in  the 
informal  phase.  System  and  acceptance  level  testing  are  done  in 
the  formal  phase.  Together,  these  four  levels  satisfy  the  Phase 
II  testing  requirements  for  the  Software  Component.  Figure  1 
depicts  software  testing  phases  and  their  levels.  The  following 
paragraphs  detail  each  test  phase  accompanied  by  the  two  levels 
of  testing  within  the  phase. 

It  is  appropriate  to  equate  the  levels  of  testing  described 
in  this  document  with  the  test  types  described  in  the  quality 
assurance  provisions  paragraphs  in  the  Development  Specifications 
for  the  Management  CPCI,  Evaluation  CPCI,  and  System  Support 
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CPCI .  Computer  Program  Test  and  Evaluation  as  described  in  the 
Development  Specifications  is  equivalent  to  unit  level  and 
integration  level  testing  as  described  in  this  document. 


PHASE:  INFORMAL 


FORMAL 


LEVEL: 


SOFTWARE 

PRODUCT: 


VERIFIED 

PACKAGE 


VERIFIED 

PROGRAM 


VERIFIED 
PROOF  AM 


DEMONSTRATED 

PROGRAM 


WHO: 


SOFTWARE  DEVELOPMENT 


NON  SOFTWARE  DEVELOPMENT 


Figure  1.  AOTS  SOFTWARE  TEST  PLAN 
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Preliminary  Qualification  Tests  as  described  in  the  Development 
Specifications  is  equivalent  to  system  level  testing  as  described 
in  this  document.  Formal  Qualification  Tests  as  described  in  the 
Development  Specifications  is  equivalent  to  acceptance  level 
testing  as  described  in  this  document. 
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2.  REFERENCED  DOCUMENTS .  The  following  government  documents 
are  referenced  in  this  plan.  These  documents  were  written  by  DAC 
as  part  of  system  design  and  development  under  the  AOTS  contract. 


70S647100 

70S647300 

70S647401 

70S647411 

70S647413 

70S647414 


Prime  Item  Development  Specif ication  for  the 

Management  Subsystem  of  the  AOTS 

Prime  Item  Development  Specification  for  the 

Evaluation  Subsystem  of  the  AOTS 

Critical  Item  Development  Specification  for  the 

Hardware  Component  of  the  AOTS 

AOTS  Management  CPCI  Development  Specif ication 

AOTS  Evaluation  CPCI  Development  Specif ication 

AOTS  System  Support  CPCI  Development  Specification 

AOTS  Software  Development  Plan 

AOTS  Configuration  Management  Plan 

AOTS  Master  Test  Plan 
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3.  PLANS  FOR  INFOiJMAL  TESTING.  The  informal  phase  of  testing 
consists  of  unit  level  testing  and  integration  level  testing.  It 
is  informal  because  the  testing  takes  place  as  the  code  is  being 
developed  and  is  planned  and  performed  by  the  people  doing  that 
development.  The  expected  result  of  informal  testing  is  a 
verified  program. 

3.1  Unit  Test  Planning.  The  intent  of  unit  testing  is  to 

verify  that  a  program  unit  performs  as  specified  in  the 

respective  CPCI  Product  Specification.  For  the  purposes  of  AOTS 
software  development,  unit  is  defined  as  an  Ada  package. 

3.1.1  Unit  Test  Requirements.  Unit  testing  consists  of  two 

phases.  The  first  phase  consists  of  compiling  the  unit  and 

correcting  compilation  errors.  The  second  phase  consists  of 
testing  the  unit  for  adherance  to  the  following  general 

requirements: 

a.  Conformation  to  specifications  and  requirements  will  be 

verified; 

-  Check  that  the  unit  is  necessary 

-  Check  function,  logic,  and  computations 

-  Check  adherance  to  standards  -  justify  exceptions 

b.  Execution  paths  will  be  verified; 

-  Execute  every  instruction  at  least  once 

-  Every  decision  should  be  executed  at  least  once  in  each 
possible  direction 

c.  Data  handling  capability  will  be  verified; 

-  Check  type  and  format  of  input  data 

-  Check  input  data  at  nominal,  extreme,  erroneous,  and 
exceptional  values 

-  Check  value,  type,  and  format  of  output  data 

d.  Design  extremes  will  be  verified. 

-  Check  error  detection  and  recovery 

This  second  phase  is  accomplished  by  desk  checking  the  unit 
and  actual  execution  of  the  unit.  The  testing  continues  until 
all  known  errors  have  been  eliminated  and  coded  unit  logic 
matches  the  design.  Interfaces  may  be  checked  by  using  stubs  to 
simulate  other  units. 

3.1.2  Unit  Test  Responsibilities.  The  unit  test  is  planned, 

performed,  and  controlled  by  the  programmer  responsible  for  the 
unit.  .  Planning  includes  what  the  programmer  intends  to 
accomplish  for  a  test,  the  inputs  that  are  required,  the  outputs 
that  are  expected,  and  how  the  test  is  to  be  conducted. 
Paragraph  3.1.4  provides  a  model  for  putting  this  data  together 
m  to  a  unit  test  procedure.  The  progx'ammer  prepares  a 
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procedure,  according  to  this  model,  that  contains  the  tests  to  be 
conducted  for  each  unit.  This  procedure  should  be  reviewed  at 
the  unit  code  walkthru.  The  procedure  is  maintained  in  the  Unit 
Development  Folder  (UDF) .  Much  of  the  testing  at  this  level  is 
very  detailed  and  will  not  be  conducted  again  in  higher  level 
tests.  Thus,  the  record  of  informal  testing  and  the 
corresponding  test  results  will  be  kept  in  the  UDF  and  preserved 
for  reference  purposes.  A  model  test  report  is  provided  in 
paragraph  3.1.5  to  assist  in  this. 

Unit  testing  can  uncover  coding  deficiencies.  The 
programmer  responsible  for  the  unit  is  expected  to  correct  the 
deficiencies  and  perform  the  unit  test  again. 

Unit  testing  can  also  uncover  design  deficiencies.  The 
programmer  responsible  for  the  unit  should  report  the  possibility 
of  any  such  conditions  to  the  CPCI  team  leader  and,  in  turn,  to 
the  software  manager.  Together,  they  will  determine  the  effect 
on  the  unit  and  released  documentation.  Correction  of  the 
deficiencies  of  this  nature  must  be  handled  in  accordance  with 
the  Configuration  Management  Plan. 

During  unit  testing,  the  primary  goal  is  to  test  the 
functions  of  the  unit.  Thus,  stubs  are  used  to  simulate  other 
units  which  the  unit  in  test  is  targeted  to  interface  with,  but 
which  are  still  under  development.  Data  that  the  unit  in  test 
provides  to  another  unit  must  be  checked  for  accuracy.  However, 
data  provided  by  another  unit  to  the  unit  in  test  could  be 
simulated. 

3.1.3  Unit  Test  Schedule.  Initial  unit  testing  takes  place 
during  the  software  development  period  for  a  unit.  Thus  the  unit 
test  schedule  follows  the  software  development  schedule  for  a 
program.  The  general  program  development  schedule  is  specified 
in  the  AOTS  Software  Development  Plan.  The  detailed  unit  test 
schedules  are  kept  with  the  test  procedures  in  the  UDFs. 

3.1.4  Unit  Test  Procedure  Model.  Attachment  A  provides  a 
model  test  procedure.  This  is  to  be  used  as  a  guide  for  the 
individual  creating  specific  AOTS  software  unit  test  procedures. 
The  following  paragraphs  describe  the  elements  of  the  procedure 
and  include  instructions  to  be  used  by  the  author  in  making  up 
the  procedure. 

Section  I  identifies  the  unit  and  the  author  of  a  test 
procedure.  Both  of  these  items,  as  well  as  the  preparation  date 
of  the  procedure,  should  be  filled  in.  The  test  type  should  be 
identified  as  "unit".  As  the  document  is  revised,  the  revision 
information,  including  author  and  date,  should  be  filled  in. 

Section  II  is  scheduling  information  for  the  performance  of 
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the  unit  test  during  development.  Enter  the  date  that  the  test 
is  projected  to  start  and  the  date  it  is  expected  to  complete,  as 
well  as  an  estimate  of  the  length  of  time  to  perform  the 
procedure.  These  entries  can  be  used  for  planning  and  scheduling 
purposes  for  this  unit.  The  development  hours  entry  is  a  record 
of  how  long  this  test  procedure  took  to  develop  and  is  intended 
to  be  used  for  future  planning  and  schedule  purposes. 

Section  III  specifies  the  plan  for  resources  necessary  for 
conducting  the  unit  test.  Identify  the  source  of  the  unit  to  be 
tested.  Identify  data  base  files,  if  any,  that  are  necessary  to 
test  this  unit.  Indicate  whether  the  files  are  new  to  this  unit 
or  if  existing  files  and  data  can  be  used.  Identify  other  units, 
if  any,  that  are  necessary  to  test  this  unit.  Besides  serving  as 
a  planning  tool  this  provides  a  low  level  critical  path,  i.e., 
the  actual  versions  of  unit  x  and  unit  y  have  to  be  completed  for 
you  to  test  unit  z.  Identify  hardware  requirements  for  the  test. 
If  the  normal  AOTS  hardware  environment  is  to  be  used,  then 
indicate  this.  If  the  unit  requires  hardware  that  has  not  been 
used  in  the  AOTS  test  environment  before,  e.g.  mark  sense  reader 
or  X/Y  input  device,  then  list  this  equipment. 

Section  IV  identifies  debug  information,  should  that  be 
appropriate.  Does  the  unit  have  special  debugging  code?  If  so, 
how  is  that  feature  activated? 


Section  V  describes  the  specific  functional  testing  to  be 
performed  on  the  unit.  Identify  the  elements  of  the  unit  that 
are  tested  by  this  procedure.  Identify  the  functions  of  the  unit 


that  are  tested  by  this  procedure.  Identify  execution  paths  in 
the  unit  that  are  not  tested  by  this  procedure.  This  may  include 
fatal  error  conditions,  nonexistant  production  data  base  files, 
permanent  i/o  errors,  etc.  Elaborate  on  the  actual  sequence  for 
the  test  procedure:  what  inputs  are  necessary,  what  outputs  are 
expected  from  that  input,  and  what  function  does  this 
input/ output  sequence  check  out  (this  is  indicated  with  an 
ordinal  cross  reference  to  the  above  function  list).  If  specific 
input  from  a  file  is  necessary  for  the  test,  indicate  what  this 
data  is.  Similarly,  record  what  the  condition  of  particular 
files  are  as  a  result  of  this  test.  List  what  error  handling  is 
to  be  verified  in  this  unit.  If  the  unit  uses  data  produced  by 
another  unit  or  produces  data  to  be  used  by  another  unit,  then 
this  information  should  be  recorded  under  "data  compatibility". 
The  heading  "other  testing"  is  to  used  to  describe  testing  that 
you  feel  should  be  performed  but  does  not  fall  into  one  of  the 


other  classes.  When  writing  the  test  procedure,  be  familiar  with 
the  contents  of  the  Test  Report  and  the  Testing  Check  List, 
described  in  the  following  paragraph,  so  that  minimum  and  maximum 
data  ranges,  as  well  as  typical  values  and  similar  constraints 
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can  be  tested.  Remember  to  author  the  test  plan  in  such  a  manner 
that  other  people,  besides  yourself,  can  perform  the  test.  Be 
clear,  complete,  yet  concise. 


3.1.5  Unit  Test  Report  Model.  Attachment  B  is  a  model  test 
report.  It  is  to  be  used  as  a  guide  to  report  on  the  execution 
of  unit  tests  and  should  be  filled  out  by  the  individual 
performing  the  test  during  the  process  of,  or  immediately  after, 
a  unit  test.  The  Test  Report  should  be  used  as  an  aid  in  all 
unit  testing,  with  the  completed  test  reports  placed  in  the  UDF. 
When  unit  testing  is  performed  at  several  levels,  i.e.  testing 
each  subprogram,  or  group  of  subprograms,  followed  by  testing  at 
the  package  level,  the  test  reporting  should  follow  that  same 
plan.  The  following  paragraphs  describe  the  elements  of  the 
report. 

Section  I  identifies  the  unit  tested,  the  person  that 
performed  the  test,  and  the  date  of  the  test.  The  same  report 
form  is  used  to  document  results  for  a  unit  level  test  and  for  an 
integration  level  test.  Therefore  mark  the  test  level  as 
appropriate.  Fill  in  the  environment  that  the  unit  was  executed 
in.  The  time  duration  of  the  test  can  be  used  for  future 
planning  purposes  in  unit  testing  and  should  be  entered  as  hours 
or  minutes. 

Section  II  is  the  reason  the  test  was  performed.  Is  the 
test  being  performed  on  a  newly  developed  unit,  is  a  unit  being 
enhanced,  or  is  the  unit  being  tested  because  other  units  that  it 
references  were  changed?  Unit  test  reports  are  necessary  not 
only  for  new  units  but  also  for  maturing  units. 

Section  III  is  an  indication  of  the  test  results.  Was  the 
outcome  expected  or  unexpected?  If  the  outcome  was  not  as 
expected,  then  a  description  of  the  problem  or  the  action  to  be 
performed  should  be  furnished,  either  in  this  section  or  in  the 
following  checklist.  This  area  can  also  be  used  for  notes  or 
reminders  if  the  test  was  successful. 

Section  IV  is  a  worksheet  to  be  used  in  conjunction  with  the 
test  sequence  section  of  the  test  procedure.  It  is  to  be  used  to 
note  problems  or  to  act  as  a  placeholder  while  executing  the  test 
sequence. 
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as  a  guide,  as  well  as  a  reminder  of  what  to  test  and  what  to 
look  for  while  testing.  The  headings  in  the  checklist  represent 
the  major  software  performance  and  requirement  classifications  to 
be  checked  in  unit  testing.  Beneath  each  heading  is  a  list  of 
items  to  exercise  or  check.  As  items  are  tested,  they  are  marked 
off  or  notes  made  for  problem  reminders. 
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3.2  CPC  Integration  and  Test  Planning.  Computer  Program 
Component  (CPC)  Integration  consists  of  building  a  program  by 
iteratively  adding  units,  and,  as  the  units  are  integrated, 

testing  to  ensure  the  resulting  software  matches  that  described 
in  the  Development  and  Product  Specifications. 

At  each  iteration,  integration  subelements  are  combined  to 
form  integration  elements.  This  process  begins  with  two  or  more 

functionally  and  logically  related  units  which  have  passed  unit 

testing.  These  units  are  integrated  and  the  resultant 
integration  element  tested.  Once  fully  tested,  such  elements 
become  the  subelements  which  are  integrated  to  form  larger 

elements.  Thus,  an  integration  subelement  may  be  either  a  fully 
tested  unit  or  a  fully  integrated  set  of  units  and  an  integration 
element  is  a  set  of  functionally  and  logically  related 
subelements. 

Ultimately,  aggregates  of  integration  elements  form  CPCs. 
For  the  purposes  of  AOTS  software  development,  the  terminology 
CPC  and  program  are  used  interchangeably. 

3.2.1  CPC  Integration  and  Test  Requirements.  Many  of  the 
requirements  for  integration  testing  are  the  same  as  for  unit 
testing.  Other  requirements  are  imposed  to  ensure  that 
subelements  are  compatible  and  consistent. 

Like  unit  testing,  integration  testing  consists  of  two 
phases.  The _  first  phase  consists  of  compiling,  linking,  and 
loading  the  integration  element  and  correcting  any  resulting 
errors.  The  second  phase  consists  of  testing  the  unit  for 
adherance  to  the  following  general  requirements: 

a.  Conformation  to  specifications  and  requirements  will  be 

verified; 

-  Check  that  the  elements  are  necessary 

-  Check  function,  logic,  and  computations 

“  Check  adherance  to  standards  -  justify  exceptions 

b.  Execution  paths  will  be  verified; 

"  Execute  every  major  path  through  the  subelement 

-  Every  subelement  should  be  called  at  least  once  by  each 
possible  calling  subelement 

c.  Data  handling  capability  will  be  verified; 

-  Check  type  and  format  of  input  data 

-  Check  input  data  at  nominal,  extreme,  erroneous,  and 
exceptional  values 

-  Check  value,  type,  and  format  of  output  data 

d.  Design  extremes  will  be  verified. 

-  Check  error  detection  and  recovery 

-  Check  data  compatibility  between  subelements 

Finally,  the  testing  must  verify  that  the  integration  is 
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forming  the  CPCs  as  they  are  described  in  the  specifications.  If 
the  CPCs  interface  with  other  CPCs,  then  these  interfaces  must 
also  be  validated. 

The  second  phase  is  accomplished  by  desk  checking  the 
element  and  actual  execution  of  the  element.  The  testing 
continues  until  all  known  errors  have  been  eliminated  and 
integrated  element  logic  matches  the  design. 

3.2.2  CPC  Integration  and  Test  Responsibilities.  As  in  unit 
testing,  the  CPC  integration  test  is  planned,  performed,  and 
controlled  by  the  programmer (s)  responsible  for  the  CPC. 
Planning  includes  what  the  programmer  intends  to  accomplish  for  a 
test,  the  inputs  that  are  required,  the  outputs  that  are 
expected,  and  how  the  test  is  to  be  conducted.  Paragraph  3.2.5 
provides  a  model  for  a  CPC  integration  test  procedure.  The 
programmer  prepares  a  procedure,  according  to  this  model,  that 
contains  the  tests  to  be  conducted  for  the  CPC.  This  procedure 
should  be  reviewed  at  the  CPC  walkthru.  This  test  procedure  is 
maintained  in  the  UDF  for  the  unit  that  is  the  linkable  entity 
that  results  in  a  program.  A  record  of  test  performance  along 
with  the  corresponding  test  results  should  be  preserved  for 
reference  purposes  in  the  UDF.  A  model  test  report  is  provided 
in  paragraph  3.2.6  to  assist  in  this. 

CPC  integration  testing  can  uncover  coding  deficiencies. 
The  programmer (s)  responsible  for  the  CPC  is  expected  to  correct 
the  deficiencies  and  perform  the  CPC  integration  test  again.  If 
the  testing  reveals  problems  with  a  unit  outside  of  the  CPC  thal 
the  CPC  is  integrating  with,  then  the  programmer  should  report 
this  to  the  CPCI  Team  Leader,  and  if  the  unit  is  outside  of  the 
particular  CPCI,  to  the  software  manager.  Resolution  for  the 
integration  deficiency  should  be  made  as  quickly  as  schedules 
allow  so  as  to  not  hamper  further  testing. 

CPC  integration  testing  can  also  uncover  design 
deficiencies.  The  programmer  responsible  for  the  CPC  should 
report  the  possbility  of  any  such  conditions  to  the  CPCI  team 
leader  and,  in  turn,  to  the  software  manager.  Together,  they 
will  determine  the  effect  on  the  CPC  and  released  documentation. 
Correction  of  the  deficiencies  of  this  nature  must  be  handled  in 
accordance  with  the  Configuration  Management  Plan. 

Any  stubs  that  were  used  in  unit  testing  are  replaced,  in 
stages,  by  actual  units  until  CPC  integration  testing  is 
complete.  The  product  of  a  completed  CPC  integration  test  is  a 
complete  CPC. 

3.2.3  CPC  Integration  Test  Classes.  The  test  classes  for  CPC 
integration  testing,  and  as  appropriate  for  unit  testing,  include 
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functional  requirement  testing,  interface  testing,  terminal  and 
device  input/output  testing,  file  input/output  testing,  and  error 
handling  testing.  The  model  test  procedure  described  in 
paragraph  3.2.5  and  the  checklist  described  in  the  model  test 
report  in  paragraph  3.2.6  reflect  these  test  classes. 

3.2.4  CPC  Integration  and  Test  Schedules.  Integration  testing 
for  a  program  takes  place  during  the  software  development  cycle 
for  that  program.  Thus  the  integration  test  schedule  follows  the 
program  development  schedule.  The  general  program  development 
schedule  is  specified  in  the  AOTS  Software  Development  Plan.  The 
detailed  integration  test  schedules  are  kept  with  the  test 
procedures  in  the  UDFs. 

3.2.5  Integration  and  Test  Procedure  Model.  Attachment  A  is  a 
model  test  procedure.  The  same  model  is  used  for  the  unit  test 
procedure  and  the  integration  and  test  procedure,  as  much  of 
required  information  is  the  same.  As  in  the  unit  test  procedure 
description,  this  model  is  to  be  used  as  a  guide  for  the 
individual  preparing  specific  integration  test  procedures.  The 
following  paragraphs  describe  the  elements  of  the  procedure  and 
include  instructions  to  be  used  by  the  author  in  formulating  the 
procedure . 

Section  I  identifies  the  CPC  to  be  integrated  and  the  author 
of  the  test  procedure.  Each  of  these  items,  as  well  as  the 
preparation  date  of  the  procedure,  should  be  completed.  The 
test  type  should  be  identified  as  "integration" .  As  the  document 
is  revised,  the  revision  information,  including  author  and  date, 
should  be  filled  in. 

Section  II  i r  scheduling  information  for  the  performance  of 
the  integration  and  test  during  development.  Enter  the  date  that 
the  test  is  projected  to  start  and  the  date  it  is  expected  to 
complete,  as  well  as  an  estimate  of  the  length  of  time  to  perform 
the  procedure.  These  entries  can  be  used  for  planning  and 
scheduling  purposes  for  this  unit.  The  development  hours  entry 
is  a  record  of  how  long  this  test  procedure  took  to  develop  and 
is  intended  to  be  used  for  future  planning  and  schedule  purposes. 

Section  III  specifies  the  plan  for  resources  necessary  for 
conducting  the  integration  and  test.  Identify  the  source  of  the 
unit  c  'ntaining  the  subelements  being  integrated.  Identify  data 
base  files,  if  any,  that  are  necessary  to  test  this  unit. 
Identify  people  whose  expertise  or  time  is  required.  Identify 
other  software,  such  as  stubs,  drivers,  or  test  data  generators, 
necessary  to  test  this  element.  Identify  unique  hardware,  i.e. 
hardware  of  limited  availability  or  hardware  not  normally  a  part 
of  the  AOTS  hardware  .^vironment,  that  is  necessary  to  test  this 
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element.  Unique  hardware  might  include  a  logic  analyzer. 
Limited  availability  hardware  might  include  a  mark  sense  reader 
or  X/Y  input  device. 

Section  IV  identifies  debug  information,  should  that  be 
appropriate.  Does  the  element  have  special  debugging  code?  If 
so,  how  is  that  feature  activated? 

Section  V  describes  the  specific  functional  testing  to  be 
performed  by  the  integration  test.  Identify  the  elements  that 
are  being  integrated  and  tested  by  this  procedure.  Identify  the 
functions  of  the  unit  that  are  tested  by  this  procedure. 
Identify  execution  paths  in  the  unit  that  are  not  tested  by  this 
procedure.  This  may  include  fatal  error  conditions,  nonexistant 
production  data  base  files,  permanent  i/o  errors,  etc.  Elaborate 
on  the  actual  sequence  for  the  test  procedure:  what  inputs  are 
necessary,  what  outputs  are  expected  from  that  input,  and  what 
function  does  this  input/output  sequence  check  out  (this  is 
indicated  with  an  ordinal  cross  reference  to  the  above  function 
list) .  If  specific  input  from  a  file  is  necessary  for  the  test, 
indicate  what  this  data  is.  Similarly,  record  what  the  condition 
of  particular  files  are  as  a  result  of  this  test.  List  what 
error  handling  is  to  be  verified  in  this  unit.  If  the  test  uses 
data  produced  by  another  test  procedure  or  produces  data  to  be 
used  by  another  test  procedure,  then  this  information  should  be 
recorded  under  "data  compatibility".  The  heading  "other  testing" 
is  to  be  used  to  describe  testing  that  you  feel  should  be 
performed  but  does  not  fall  into  one  of  the  other  classes.  When 
writing  the  test  procedure,  be  familiar  with  the  contents  of  the 
Test  Report  and  its  Testing  Check  List,  described  in  the 
following  paragraph,  so  that  minimum  and  maximum  data  ranges,  as 
well  as  typical  values  and  similar  constraints  can  be  tested. 

Section  V  of  this  test  plan  is  much  like  Section  V  of  the 
Unit  Test  Plan.  However,  the  intent  of  the  Specific  Testing 
section  is  different  between  the  two  plans.  In  the  Unit  Test 
Plan  the  scope  of  the  Specific  Testing  section  deals  with  all 
instructions  within  the  unit.  In  the  Integration  and  Test  Plan, 
the  Specific  Testing  section  is  concerned  with  major  paths 
through  elements  and  all  of  the  calling  sequences  of  subelements 
being  integrated. 

3.2.6  Integration  and  Test  Report  Model.  Attachment  B  is  a 
model  test  report.  As  with  the  test  procedure  model,  the  same 
model  for  reports  is  used  for  unit  testing  and  integration 
testing,  as  much  of  required  information  is  the  same.  As 
described  in  the  unit  test  report  description,  this  model  is  to 
be  used  as  a  guide  to  report  on  the  execution  of  integration 
tests  and  should  be  filled  out  by  the  individual  performing  the 
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test  during  the  process  of,  or  immediately  after,  an  integration 
test.  The  Test  Report  should  be  used  as  an  aid  in  all 
integration  and  testing,  with  the  completed  test  reports  placed 
in  the  UDF.  The  following  paragraphs  describe  the  elements  of 
the  report. 

Section  I  identifies  the  integration  test  being  performed, 
the  person  that  performed  the  test,  and  the  date  of  the  test. 
The  same  report  form  is  used  to  document  results  for  a  unit  level 
test  and  for  an  integration  level  test.  Therefore  mark  the  test 
level  as  appropriate.  Fill  in  the  environment  that  the  test  was 
executed  in.  The  time  duration  of  the  test  can  be  used  for 
future  planning  purposes  in  integration  testing  and  should  be 
entered  as  hours  or  minutes. 

Section  II  is  the  reason  the  test  was  performed.  Is  the 
test  being  performed  on  a  newly  developed  element,  is  an  element 
or  subelement  being  enhanced,  or  is  the  element  being  tested 
because  other  elements  that  it  references  were  changed? 
Integration  and  test  reports  are  necessary  not  only  for  newly 
integrated  elements  but  also  for  maturing  elements. 

Section  III  is  an  indication  of  the  test  results.  Was  the 
test  outcome  expected  or  unexpected?  If  the  outcome  was  not  as 
expected,  then  a  description  of  the  problem  or  the  action  to  be 
performed  should  be  furnished,  either  in  this  section  or  in  the 
following  checklist.  This  area  can  also  be  used  for  notes  or 
reminders  if  the  test  results  were  satisfactory. 

Section  IV  is  a  worksheet  to  be  used  in  conjunction  with  the 
test  sequence  section  of  the  test  procedure.  It  is  to  be  used  to 
note  problems  or  to  act  as  a  placeholder  while  executing  the  test 
sequence. 

Section  V  is  a  checklist  for  integration  and  testing.  It  is 
to  be  used  as  a  guide,  as  well  as  a  reminder  of  what  to  test  and 
what  to  look  for  while  testing.  The  headings  in  the  checklist 
represent  the  major  software  performance  and  requirement 
classifications  to  be  checked  in  unit  testing.  Beneath  each 
heading  is  a  list  of  items  to  exercise  or  check.  As  items  are 
tested,  they  are  marked  off  or  notes  made  for  problem  reminders. 

3.3  Resources  Required  for  Informal  Testing. 

3.3.1  Facilities.  The  DAC  facilities  at  Building  428 
Bergstrom  AFB  Austin,  Texas,  will  be  the  location  at  which 
informal  testing  will  be  performed.  The  AFHRL  facilities  at 
Building  578  Brooks  AFB  San  Antonio,  Texas,  will  be  the  location 
where  the  host  computer  for  AOTS  resides.  No  classified 
information  will  be  processed  in  conjunction  with  the  AOTS  Phase 
II  informal  tests. 
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3.3.2  Personnel.  Personnel  required  for  informal  testing  are 
members  of  the  DAC  AOTS  Software  Development  Organization.  An 
in-depth  knowledge  of  the  requirements  and  design  of  the  unit  or 
CPC  whose  test  is  being  planned  or  performed  is  mandatory  for  the 
personnel.  Access  to  Bergstrom  AFB  is  necessary  for  these 
personnel,  however,  security  clearances  are  not  necessary. 

3.3.3  Hardware.  Hardware  to  be  used  for  informal  testing  is 
as  follows: 

A.  VAX  8600  located  at  Building  578,  Brooks  AFB; 

B.  Zenith  Z248  Personal  Computers  located  at  Building  428, 
Bergstrom  AFB; 

C.  Printers,  of  the  following  types,  located  in  Building 
428,  Bergstrom  AFB: 

1.  Laser  printers 

2.  Color  printers 

3.  Dot  matrix  printers; 

D.  Digitizing  tablets,  of  the  following  types,  located  at 
Building  428,  Bergstrom  AFB; 

1.  11x11  digitizers 

2.  20x20  digitizers; 

E.  Optical  Mark  Readers  located  at  DAC,  Building  428, 
Bergstrom  AFB; 

F.  Communication  lines  and  equipment  to  link  the  above 
hardware. 

3.3.4  Interfacing/Support  Software.  Software  required  for 
informal  testing  is  as  follows: 

A.  VAX/VMS  operating  system,  including  the  utilities: 

1 .  Command  Language 

2.  EDT 

3 .  Linker 

4 .  Debug 

5.  Run-time  Library 

B.  DEC  Ada  Compiler 

n  VAV-1  1  PADfTDAM  _ 

-*•  i  Will  J _ 

D.  VAX-11  MACRO  Assembler 

E.  DEC  Code  Management  System 

F.  MS-DOS,  including  the  utilities 

1 .  Command  Language 

2.  Editor 

3 .  Debug 

G.  Alsys  Ada  Compiler 

H.  Microsoft  Macro  Assembler 
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3.3.5  Source.  The  above  resources  are  provided  by  AFHRL  and 
DAC  in  support  of  the  AOTS  contract. 

3.3.6  Test  Configuration.  The  test  configuration  is 
graphically  portrayed  in  Figure  2.  It  consists  of  the  hardware 
iisted  in  paragraph  3.3.3,  located  at  the  facilities  specified  in 
paragraph  3.3.1.  Note  that  the  peripheral  equipment  layout  in 
Building  428  at  Bergstrom  AFB  shows  all  the  different  types  of 
devices  to  be  used  in  Phase  III  AOTS.  A  particular  unit  or 
integration  test  will  use  a  subset  of  this  equipment.  However, 
the  unit  and  integration  tests,  considered  as  a  group,  will  use 
all  of  the  devices. 
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Figure  2 
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4.  PLANS  FOR  FORMAL  TESTING.  The  formal  phase  of  testing 
consists  of  system  level  testing  and  acceptance  level  testing. 
The  objective  is  verification  that  software  meets  performance  and 
interface  requirements  and  demonstration  that  the  software  meets 
the  acceptance  criteria.  Formal  testing  is  planned  by  the 
software  development  group  but  is  performed  by  a  group 
independent  of  the  software  developers.  The  expected  product  is 
a  demonstratable  and  deliverable  program. 

Formal  testing  will  be  conducted  in  accordance  with  the 
phased  software  development  approach  for  AOTS  Phase  II.  A  formal 
test  procedure,  at  the  system  test  level,  will  be  prepared  during 
the  development  period  for  that  program.  This  test  procedure 
will  be  performed  and  results  recorded  before  the  program  is 
released  for  Air  Force  use.  Thus,  before  the  data  development 
programs  are  released  to  the  1ST,  they  will  go  through  the  system 
testing  cycle. 

Formal  test  procedures,  at  the  acceptance  test  level,  will 
be  performed  and  results  recorded,  in  order  for  the  Air  Force  to 
accept  AOTS  Phase  II  software. 

Both  system  and  acceptance  test  procedures  can  be  used  to 
satisfy  test  objectives  in  the  AOTS  Master  Test  Plan  that  require 
automated  support. 

4.1  System  Test  Planning.  The  intent  of  System  Testing  is  to 
formally  verify  software  performance  and  interface  requirements. 
System  level  testing  is  performed  on  programs  that  have  completed 
integration  level  testing. 

4.1.1  System  Test  Requirements.  System  Testing  consists  of 
tests  and  analyses  performed  to  confirm  that  the  software 
satisfies  all  the  requirements  set  forth  in  the  AOTS  CPCI 
Development  Specifications.  It  verifies  that  the  "as  built" 
software  conforms  to  these  specifications.  Formal  test  plans  and 
procedures  are  written  for  System  Testing.  Analysis  is  performed 
for  those  requirements  that  are  too  expensive,  in  terms  of  time 
or  resources,  to  verify  by  testing. 

In  order  for  a  program  to  be  ready  for  system  testing,  it 
must  have  successfully  completed  integration  level  testing  and  be 
placed  under  configuration  control.  The  system  tests  are 
performed  according  to  test  procedures.  Test  results  are 
documented,  including  any  discrepancies  found. 

4.1.2  System  Test  Responsibilities.  The  system  test 
procedures  are  prepared  by  the  DAC  AOTS  software  development 
organization,  in  particular  by  the  programmer (s)  responsible  for 
individual  CPCs.  This  preparation  is  done  during  the  software 
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development  cycle  for  a  particular  program.  The  test  procedure 
should  define  in  detail  the  actual  activities  required  to  perform 
the  test.  It  should  include  a  description  of  what  should  be 
accomplished  by  the  test,  the  inputs  that  are  required,  the 
outputs  that  are  expected,  and  how  the  test  is  to  be  conducted. 
The  programmer  responsible  for  a  CPC  will  ask  that  both  the 
program  and  test  procedure  be  placed  under  configuration 
management  when  the  program  has  completed  integration  testing, 
thereby  declaring  the  program  is  ready  for  system  testing . 

Conduct  of  the  system  tests  will  be  carried  out  by  the  DAC 
AOTS  Instructional  Technology  Branch  personnel.  These 
individuals  are  different  from  the  people  that  comprise  the 
software  development  group,  but  they  are  very  familiar  with 
system  and  program  requirements.  The  people  who  conduct  a  system 
test  will  also  be  responsible  for  recording  the  test  results. 

Monitoring  of  system  tests  will  be  performed  by  the  DAC  AOTS 
Program  Manager  or  designee.  This  person's  responsibility  will 
be  to  spot  check  that  the  system  tests  are  being  conducted 
according  to  procedures  and  that  results  are  recorded. 

The  Air  Force  AOTS  Program  Manager,  or  representative,  may 
witness  system  tests. 

Results  of  the  system  tests  are  recorded  in  test  reports.  If 
the  tests  uncovered  deficiencies,  these  are  also  recorded  in  the 
test  reports.  All  deficiencies  found  in  system  testing  are 
examined  by  the  Software  Technical  Review  Board.  This  board  is 
comprised  of  the  software  manager  and  each  CPCI  team  leader.  The 
board  decides  the  extent  of  each  deficiency  and  schedules  the 
problem  to  be  fixed  or  directs  the  deficiency  and  its 
ramifications  to  the  Configuration  Control  Board. 

4.1.3  System  Test  Schedule.  The  system  test  schedule  is 
specified  in  the  AOTS  Software  Development  Plan. 

4.1.4  System  Test  Procedure  Model,  Attachment  C  provides  a 
model  formal  test  procedure.  The  model  is  to  be  used  as  a  guide 
for  the  individual  creating  specific  formal  test  procedures  for 
AOTS  software.  Note  that  the  model  is  very  similar  to  the 
informal  testing  model  discussed  under  unit  level  testing  and 
integration  level  testing.  Also  note  that  this  formal  test 
procedure  model  is  valid  for  both  System  Test  Procedures  and 
Acceptance  Test  Procedures.  The  following  paragraphs  describe 
the  elements  of  the  procedure  and  include  instructions  to  be.  used 
by  the  author  in  making  up  the  procedure. 

Section  I  identifies  the  type  of  formal  test  procedure,  full 
identification  of  the  program  to  be  tested,  and  the  author  of  the 
test  procedure.  As  the  plan  is  revised,  the  revision  author  and 
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date  should  be  filled  in. 

Section  II  is  scheduling  information  for  the  performance  of 
the  formal  test.  Enter  a  projection  of  the  date  when  the  test 
will  start  and  how  long  will  it  take.  Also  enter  the  time 
necessary  to  develop  the  plan.  This  entry,  as  well  as  the 
projected  test  performance  date  and  duration  are  for  planning  and 
scheduling  purposes. 

Section  III  specifies  the  resources  required  for  conducting 
the  formal  test.  The  software  resource  category  identifies  the 
program  executable  to  be  tested  and  other  programs  that  are 
necessary  for  the  test.  Identify  the  data  base  files  required  to 
test  this  program.  Indicate  what  the  state  of  these  files  should 
be,  e.g.  whether  these  files  are  required  to  contain  a  previously 
determined  set  of  data,  whether  they  should  be  empty,  etc.  Some 
test  procedures  may  require  a  known  data  base  to  predict  known 
test  results.  This  resource  category  should  reflect  that. 
Identify  what  other  test  procedures  need  to  be  run  prior  to  this 
procedure.  This  is  important  when  test  procedures  need  to  be  run 
in  a  certain  order.  For  example,  it  may  be  necessary  to  run  the 
Tenative  MTI.  Test  Procedure  prior  to  running  the  MTL  Editor  -Test 
Procedure.  Identify  the  hardware  configuration  for  the  test  by 
listing  the  required  hardware.  If  the  tester  needs  special 
qualifications  to  run  the  test,  enter  this  data  under  the 
personnel  requirements  category.  An  example  of  this  might  be 
that  the  tester  should  familiar  with  one  of  the  AOTS  supported 
Air  Force  Specialities.  When  special  resoux*ces  are  required, 
such  as  a  video  disk  or  a  drawing,  enter  this  information  under 
the  "other”  category. 

Section  IV  describes  the  test.  A  high  level  overview  of  the 
test  should  be  entered  to  explain  the  test.  The  functions 
provided  by  the  program  that  the  test  procedure  is  to  check 
should  be  listed.  If  there  are  program  functions  not  tested  by 
the  procedure,  then  these  should  be  listed  together  with  the 
identification  of  the  test  procedure  in  which  they  are  tested. 

Section  V  lists  the  actions  necessary  to  perform  the  test. 
The  test  sequence  listed  here  details  the  input  expected  from  the 
user,  what  output  can  be  expected ,  and  a  cross  reference  as  to 
what  program  function  that  this  action  applies  to.  The  listed 
test  sequence  should  be  as  exact  as  possible,  in  order  to  produce 
test  consistency. 

Attachment  F  is  an  example  of  an  AOTS  Formal  Test  Procedure 
to  perform^  system  testing  on  the  task  information  field  editing 
functions  in  the  MTL  Editor.  This  is  to  serve  as  an  example  only 
and  is  not  intended  to  be  complete. 

4.1.5  System  Test  Report  Model.  Attachment  D  is  a  model  AOTS 
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Formal  Test  Report.  It  is  to  be  used  as  a  guide  to  report  on  the 
execution  of  formal  system  tests  and  should  be  filled  out  by  the 
individual  performing  the  test  during  the  process  of,  or 
immediately  after,  a  system  test. 

Section  I  identifies  the  program  tested  and  the  person  that 
did  the  test.  The  came  report  form  is  used  to  document  results 
for  a  system  level  test  and  for  an  acceptance  level  test. 
Therefore  mark  the  test  level  as  appropriate.  Fill  in  the  test 
environment,  as  well  as  the  date.  The  time  duration  of  the  test 
can  be  used  for  future  planning  purposes  in  system  testing  and 
should  be  entered  as  hours  or  minutes.  If  there  is  a  test 
witness  then  the  identity  of  that  individual  should  be  recorded. 

Section  II  is  the  reason  the  test  was  performed.  Is  the 
test  being  performed  on  a  newly  developed  program,  for  a  program 
that  has  been  enhanced,  or  as  part  of  a  release  of  other 
enhancements? 

Section  III  is  an  indication  of  the  test  results.  Was  the 
outcome  expected  or  unexpected?  If  the  outcome  was  not  as 
expected,  then  a  description  of  the  problem  or  the  action  to  be 
performed  should  be  furnished,  either  in  this  section  or  in  the 
following  worksheet.  This  area  can  also  be  used  for  notes  or 
reminders  if  the  test  was  successful. 

Section  IV  is  a  worksheet  for  system  testing .  It  is  to  be 
used  to  keep  notes  of  test  progress,  as  well  as  to  keep  track  of 
possible  discrepancies  or  questions. 

4.2  Acceptance  Test  Planning.  The  intent  of  acceptance 
testing  is  to  demonstrate  that  the  software  satisfies  the  set  of 
predetermined  acceptance  criteria. 

4.2.1  Acceptance  Test  Requirements.  Acceptance  Testing 
consists  of  performing  test  procedures  designed  to  confirm  that 
the  software  satisfies  all  the  requirements  set  forth  in  the 
AOTS  CPCI  Development  Specifications.  Acceptance  testing  must 
show  that  the  software  performs  in  the  operational  environment, 
as  it  did  in  test  environment.  This  includes  using  actual 
external  interfaces. 

Generally  acceptance  testing  is  performed  when  a  system  is 
delivered.  In  the  case  of  AOTS,  acceptance  testing  will  be 
performed  during  the  latter  part  of  Phase  II.  Test  results  are 
documented,  including  any  discrepancies  found.  Some 
discrepancies  are  permissible.  The  Air  Force  and  DAC  AOTS 
management  must  agree  jointly  upon  the  number  and  severity  of 
permissible  discrepancies. 

4.2.2  Acceptance  Test  Responsibilities.  The  acceptance  test 
procedures  are  prepared  by  the  DAC  AOTS  software  development 


AOTS . STP 


D-  20 


VI. 0/Feb. 87 


^mast- 


organization,  in  particular  by  the  programmer (s)  responsible  for 
individual  CPCs.  This  preparation  is  done  after  system  testing 
has  been  performed  on  the  CPC.  The  test  procedure  should  define 
in  detail  the  actual  activities  required  to  perform  the  test.  It 
should  include  a  description  of  what  should  be  accomplished  by 
the  test,  the  inputs  that  are  required,  the  outputs  that  are 
expected,  and  how  the  test  is  to  be  conducted.  The  acceptance 
test  procedure  is  placed  under  configuration  control  after  it  is 
written  and  approved. 

Conduct  of  the  acceptance  tests  will  be  carried  out  by  the 
DAC  AOTS  Instructional  Technology  Branch  personnel.  These 
individuals  may  be  the  same  people  that  performed  .the  system 
tests.  They  should  be  familiar  with  system  and  program 
requirements.  The  people  who  conduct  an  acceptance  test  will 
also  be  responsible  for  recording  the  test  results. 

Monitoring  of  acceptance  tests  will  be  performed  by  the  DAC 
AOTS  Program  Manager  or  designee.  This  person's  responsiblity 
will  be  to  verify  the  acceptance  tests  are  being  conducted 
according  to  procedures  and  that  results  are  recorded. 

The  Air  Force  AOTS  Program  Manager,  or  representative,  will 
witness’  acceptance  tests. 

Results  of  the  acceptance  tests  are  recorded  in  test 
reports.  If  the  tests  uncover  deficiencies,  these  are  also 
recorded  in  the  test  reports.  As  in  system  testing,  all 
deficiencies  found  in  acceptance  testing  are  examined  by  the 
Software  Technical  Review  Board.  The  board  decides  the  extent  of 
each  deficiency  and  schedules  the  problem  to  be  fixed  or  directs 
the  deficiency  and  its  ramifications  to  the  Configuration  Control 
Board. 

Execution  of  an  acceptance  test  procedure  with  expected  and 
agreed  upon  results,  as  well  as  the  generation  of  the 
corresponding  test  report,  should  result  in  acceptance  of  the 
program  by  the  Air  Force  AOTS  Program  Manager. 

4.2.3  Acceptance  Test  Schedule.  The  acceptance  test  schedule 
is  specified  in  the  AOTS  Software  Development  Plan. 

4.2.4  Acceptance  Test  Procedure  Model.  Attachment  C  provides 
a  model  formal  test  procedure.  The  model  is  discussed  in 
paragraph  4.1.4.  Sections  I,  II,  III,  and  IV  of  the  test 
procedures  should  be  filled  out  accordingly. 

Section  V  of  the  acceptance  test  procedure  has  the  same 
format  as  the  system  test  procedure.  This  section,  the  test 
sequence,  should  list  the  actions  necessary  to  perform  the  test. 
The  test  sequence  includes  the  input  expected  from  the  user,,  what 
output  can  be  expected,  and  a  cross  reference  as  to  what  program 
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function  that  the  actions  apply  to.  However,  the  inputs  and 
outputs  listed  in  the  acceptance  test  procedure  are  more 
descriptive  than  in  the  system  test  procedure.  The  reason  for 
this  is  that  since  the  acceptance  test  procedures  can  be  used  for 
demonstration  purposes,  the  audience  for  the  test  procedure  might 
not  be  familiar  with  the  program  being  demonstrated. 

Attachment  G  is  an  example  of  an  AOTS  Formal  Test  Procedure 
to  perform  acceptance  testing  on  the  MTL  Editor.  This  is  to 
serve  as  an  example  only  and  is  not  intended  to  be  complete. 

4.2.5  Acceptance  Test  Report  Model.  Attachment  D  is  a  model 
AOTS  Formal  Test  Report.  It  is  to  be  used  as  a  guide  to  report 
on  the  execution  of  formal  acceptance  tests  and  should  be  filled 
out  by  the  individual  performing  the  test  during  the  process  of, 
or  immediately  after,  an  acceptance  test.  The  sections  of  the 
model  are  discussed  further  in  paragraph  4.1.5. 

4.3  Formal  Test  Classes.  The  test  classes  for  formal  testing 
include  functional  requirement  testing,  interface  testing,  user 
input/cutput  testing,  file  input/ output  testing,  error  condition 
testing,  timing  constraints,  and  capacity  testing.  The  first  six 
of  these  classes  are  verified  by  each  system  and  acceptance  test 
procedure  for  which  the  class  is  appropriate.  AOTS  capacity 
testing  is  verified  by  its  own  test  procedure. 

4.4  Formal  Tests.  Each  AOTS  CPCI  will  have  a  series  of  formal 
tests,  both  at  the  system  and  acceptance  test  level,  designed  to 
verify  each  CPC  and  interface  required  by  the  CPCI.  The  tests 
will  be  structured  according  to  the  models  described  in 
paragraphs  4.1.4  and  4.2.4..  The  paragraphs  below  identify  these 
tests  for  the  . initial  design  and  development  effort.  Further 
tests  will  be  identified  during  the  secondary  design  effort  with 
an  update  to  this  document  prior  to  the  Critical  Design  Review 
for  the  total  software  effort. 


4.4.1  Management  CPCI  Test  Procedures.  The  Management  CPCI 
Test  Procedures  will  consist  of  the  tests  listed  in  Table  1. 
with  each  entry  is  a  paragraph  reference  to  the  Management  CPCI 

DftVPl  OniTlPnf'  Cricw^i  f  ■!  n  ^v»  mu  »  ^  ^  •  .  • 

^ -  — ub  reference  specines  me 

functions  to  be  performed  and  the  requirements  to  be  satisfied  by 
the  program  and  therefore  to  be  tested  by  the  test  procedure. 
The  actual  test  procedure  will  describe  the  test,  the  functions 
the  procedure  is  to  check,  and  the  inputs  and  expected  outputs  to 
exercise  the  functions.  Other  entries  in  the  table  specify  the 

.  testmg  to  be  performed  for  each  program  and  the  test 
methods  involved  to  check  each  program. 
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Table  1.  Management  CPCI  Test  Procedures 


Test  Test 

£ _ Procedure  Name _ 

1  Tenative_MTL 

2  Final_MTL 

3  MTL_Editor 

4  Task_Fublications_Editor 

5  Generic_Position_Training_ 
Requirements_Editor 

6  Operational_Position__ 
Tra.ining_Requirements_Editor 

7  Otner_Training__Requirements  _ 
Editor 

8  IMEdit 

9  Requirement  Deleted 

10  Requirement  Deleted 

11  Capacity  Test 

12-n  (4) 


70S647411 

Test 

Test 

Ref  n )  Level ( 2)  Me 

;thod 

3. 2. 1.1.1 

U , I , S , Ac 

T,D 

3. 2. 1.1. 2 

U , I , S , Ac 

T,  D 

3. 2. 1.1. 3 

U , I , S , Ac 

T,  D 

3. 2. 1.2.1 

U ,  I , S , AC 

T,  D 

3. 2. 1.4.1 

U ,  I , S , AC 

T,  D 

3. 2. 1.4. 2 

U,  I,S,AC 

T,  D 

3.2. 1.5 

U ,  I , S , Ac 

T,  D 

3 . 2 . 3 . 1 . 1 

U  ,  I ,  S  ,  Ac, 

T,  D 

3.1.1 

S ,  Ac 

T,  D 

Notes : 

1.  70S647411  is  the  AOTS  Management  CPCI  Development 
Specification.  Items  in  this  column  are  references  to 
paragraphs  in  this  specification. 

2.  Test  Level  Codes:  U=Unitr  I=Integration ,  S=System, 
Ac=Aceeptance 

3.  Test  Method  Codes:  A=Analysis,  I=Inspection,  D=Demonstration, 
T=Test,  R=Review 

4.  The  remainder  of  this  table  will  be  developed  in  accordance 
with  the  AOTS  phased  software  development  concept. 
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4.4.2  Evaluation  CPCI  Test  Procedures.  The  Evaluation  CPCI 
Test  Procedures  will  consist  of  the  tests  listed  in  Table  2. 
With  each  entry  is  a  paragraph  reference  to  the  Evaluation  CPCI 
Development  Specification.  This  reference  specifies  the 
functions  to  be  performed  and  the  requirements  to  be  satisfied  by 
the  program  and  therefore  to  be  tested  by  the  test  procedure. 
The  actual  test  procedure  will  describe  the  test,  the  functions 
the  procedure  is  to  check,  and  the  inputs  and  expected  outputs  to 
exercise  the  functions.  Other  entries  in  the  table  specify  the 
levels  of  testing  to  be  performed  for  each  program  and  the  test 
methods  involved  to  check  each  program. 


Table  2.  Evaluation  CPCI  Test  Procedures 


Test 

Test 

70S647413 

Test 

Test 

# 

Procedure  Name 

Ref  m 

Level ( 2)  Method 

1 

Behavioral_Ob j  ect ives_Editor 

3.2.1. 1 

U ,  I ,  S ,  Ac, 

T,  D 

2 

Test_Item_Bank_Editor 

3.2.1. 2 

U, I,S,Ac 

T,  D 

3 

Test_Editor 

3.2. 1.3 

U, I,S,Ac 

T,  D 

4 

Requirement  Deleted 

5 

Graph ics_Edi tor 

3.2. 1.5 

U, I,S, Ac 

T,  D 

6 

Requirement  Deleted 

7-n 

(4) 

Notes : 

1.  7 0S647413  is  the  AOTS  Evaluation  CPCI  Development 
Specification.  Items  in  this  column  are  references  to 
paragraphs  in  this  specification. 

2.  Test  Level  Codes:  U-Unit,  I=Integration,  S=System, 
Ac=Acceptance 

3.  Test  Method  Codes:  A=Analysis,  I=Inspection,  D=Demonstration, 
T=Test,  R=Review 

4.  The  remainder  of  this  table  will  be  developed  in  accordance 
with  the  AOTS  phased  software  development  concept. 
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4.4.3  System  Support  CPCI  Test  Procedures.  Many  of  the 
functional  requirements  of  the  System  Support  CPCI  are  considered 
tested  as  a  result  of  the  software  development  process  and 
informal  and  formal  testing  of  the  Management  and  Evaluation 
CPCIs.  Therefore  these  requirements  need  no  separate  test 
procedures.  These  requirements  are  listed  ir  Table  3  without  a 
test  procedure  number,  but  instead  a  reference  to  note  4.  The 
System  Support  Test  Procedures  will  test  the  remaining 
requirements  of  the  System  Support  CPCI.  The  test  procedures 
listed  in  Table  3  with  a  test  procedure  number  identify  these 
procedures.  With  each  entry  in  Table  3  is  a  paragraph  reference 
to  the  System  Support  CPCI  Development  Specification.  This 
reference  specifies  the  functions  to  be  performed  and  the 
requirements  to  be  satisfied  by  the  program  and  therefore  to  be 
tested  by  the  test  procedure,  when  a  formal  test  procedure  is 
applicable.  The  actual  test  procedure  will  describe  the  test, 
the  functions  the  procedure  is  to  check,  and  the  inputs  and 
expected  outputs  to  exercise  the  functions.  Other  entries  in  the 
table  specify  the  levels  of  testing  to  be  performed  for  each 
program  and  the  test  methods  involved  to  check  each  program. 
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Table  3.  System  Support  CPCI 

Test  Procedures 

Test 

Test 

70S647414  Test 

Test 

$ 

Procedure  Name 

Ref  (1)  Level (2)  Method ( 3) 

(4) 

Operating  System 

3. 2. 1.1 

D 

(4) 

Ada  Compiler 

3. 2. 1.2.1 

D 

(4) 

Host  Language  Compiler 

3. 2. 1.2. 2 

D 

(4) 

Text  Editor 

3. 2. 1.3 

D 

(4) 

Virtual  Machine  Interface 

3.2.2 

D 

(4) 

Program  Control 

3. 2. 3.1 

D 

(4) 

Inter-Process  Communication 

3. 2. 3. 2 

D 

(4) 

Data  Management 

3. 2. 3. 3 

D 

(4) 

Terminal  Communication 

3. 2. 3. 4 

D 

(4) 

Text  Handling 

3. 2. 3. 5 

D 

(4) 

Mathematical  Services 

3 . 2 . 3 . 6 

D 

(4) 

AOTS  Utilities 

TBD 

D 

(4) 

Operating  System  (Terminal) 

3.2.5. 1 

D 

(4) 

Ada  Compiler  (Terminal) 

3. 2. 5. 2 

D 

1 

Terminal 

3. 2. 5. 3  U , I , S , Ac 

T,  D 

2 

Capacity  Test 

3.1.1  S , Ac 

T,  D 

3-n 

(5) 

Notes: 

1.  7 0S6474 14  is  the  AOTS  System  Support  CPCI  Development 
Specification.  Items  in  this  column  are  references  to 
paragraphs  in  this  specification. 

2.  Test  Level  Codes:  U=Unit,  I=Integration,  S=System, 
Ac=Acceptance 

3.  Test  Method  Codes:  A=Analysis,  I=Inspection,  D=Demonstration, 
T=Test,  R=Review 

4 .  These  functions  are  demonstrated  to  be  performing  properly  as 
a  result  of  other  system  and  acceptance  test  procedures  using 
the  functions. 

5.  The  remainder  of  this  table  will  be  developed  in  accordance 
with  the  AOTS  phased  software  development  concept. 
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4.5  Formal  Test  Levels.  The  levels  of  formal  testing  for  AOTS 
are  system  level  testing  and  acceptance  level  testing.  These  are 
discussed  in  paragraphs  4.1  and  4.2,  respectively. 

4.6  Formal  Test  Summary.  Identification  of  formal  tests  is 
provided  in  Tables  1,  2,  and  3  above.  Additionally,  these  tables 
identify  test  levels,  test  methods,  and  program  requirements. 
Test  classes  are  identified,  and  checked,  as  necessary  in  each 
test  procedure,  with  the  exception  of  capacity  testing,  which  has 
its  own  formal  test  procedure. 

4.7  Formal  Test  Schedule(s).  Refer  to  paragraph  4.1.3  for  the 
System  Test  Schedule  and  paragraph  4.2.3  for  the  Acceptance  Test 
Schedule. 

4.8  Data  Recording,  Reduction,  Analysis.  The  specification  of 
data  to  be  recorded,  reduced,  or  analyzed  in  order  for  programs 
to  be  properly  verified  will  be  detailed  in  the  system  and 
acceptance  test  procedures  for  those  programs.  When  the  results 
of  the  data  recording,  reduction,  or  analysis  is  on  hard  copy, 
then  this  hard  copy  will  be  stored  with  the  test  reports,  which 
are  described  in  paragraphs  4.1.5  and  4.2.5. 

A  test  log  shall  be  kept  when  performing  acceptance  test 
procedures.  All  significant  events  will  be  reported 
chronologically  on  the  test  log.  A  completed  test  log  is  stored 
with  the  test  report.  A  model  test  log  is  contained  in  Appendix 
£  • 


4.9  Formal  Test  Reports.  Formal  test  report  models  are 
described  in  paragraphs  4.1.5  and  4.2.5. 

4.10  Resources  Required  for  Formal  Testing. 

4.10.1  Facilities.  The  DAC  facilities  at  Building  428  and  the 
AFHRL  1ST  facilities  at  Building  1808,  both  at  Bergstrom  AFB 
Austin,  Texas,  will  be  the  locations  at  which  formal  testing  will 
be  performed.  The  AFHRL  facilities  at  Building  587  Brooks  AFB 
San  Antonio,  Texas,  will  be  the  location  where  the  host  computer 


J-UJ.  1  O  L  tSS>JLW«SS>  . 


*'**'-*  icu  inxuimciuiGu  wxll  be  processed  in 


conjunction  with  the  AOTS  Phase  II  formal  tests. 


4.10.2  Personnel.  Personnel  required  for  the  preparation  of 
formal  test  procedures  are  members  of  the  DAC  AOTS  Software 
Development  Organization.  An  in-depth  knowledge  of  the 
requirements  and  design  of  the  CPC  whose  test  is  being  planned  is 
mandatory  for  the  personnel. 
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Personnel  required  for  conducting  formal  tests  are  members 
of  the  DAC  AOTS  Instructional  Technology  Branch.  The  personnel 
represent  the  Management  Subsystem,  Evaluation  Subsystem,  and  1ST 
Support  group  and  must  have  the  following  qualifications: 

A.  An  in-depth  knowledge  of  one  or  all  subsystems; 

B.  Familiarity  with  the  user  interface  to  automated 
functions  for  a  subsystem; 

C.  Understanding  of  data  contents  requirements  for  the 
automated  functions . 

Access  to  Bergstrom  AFB  is  necessary  for  both  groups  of 
personnel,  however,  security  clearances  are  not  necessary. 

4.10.3  Hardware.  Hardware  to  be  used  for  formal  testing  is  as 
follows: 

A.  VAX  8600  located  at  Building  578,  Brooks  AFB; 

B.  Zenith  Z248  Personal  Computers  located  at  Buildings  428 
and  1808,  Bergstrom  AFB; 

C.  Printers,  of  the  following  types,  located  in  Buildings 
428  and  1808,  Bergstrom  AFB: 

1.  Laser  printers 

2 .  Color  printers 

3.  Dot  matrix  printers; 

D.  Digitizing  tablets,  of  the  following  types,  located  at 
Buildings  428  and  1808,  Bergstrom  AFB; 

1.  11x11  digitizers 

2.  20x20  digitizers; 

E.  Optical  Mark  Readers  located  in  Buildings  428  and  1808, 
Bergstrom  AFB; 

F.  Communication  lines  and  equipment  to  link  the  above 
hardware . 


4.10.4  Interfacing/Support  Software.  Software  required  for 
formal  testing  is  as  follows: 

A.  VAX/VMS  operating  system,  including  the  utilities: 

1.  Command  Language 

2.  EDT 

3 .  Linker 

4 .  Debug 

^  Onn—f-J  wa  T  {Uv'tvtf 
-*'  ^  jr 

B.  DEC  Ada  Compiler 

C.  VAX-11  FORTRAN  Compiler 

D.  VAX-11  MACRO  Assembler 

E.  DEC  Code  Management  System 

F.  MS-DOS,  including  the  utilities 

1 .  Command  Language 

2.  Editor 
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3 .  Debug 

G.  Alsys  Ada  Compiler 

H.  Microsoft  Macro  Assembler 

4.10.5  Source.  The  above  resources  are  provided  by  AFHRL  and 
DAC  in  support  of  the  AOTS  contract. 

5.  TEST  PLANNING  ASSUMPTIONS  AND  CONSTRAINTS.  To  help  assure 
valid  system  level  and  acceptance  level  testing,  it  is  assumed 
that  actual  AOTS  data,  e.g.  Master  Task  List  data.  Behavioral 
Objectives,  Test  Items,  etc.,  will  be  available  to  exercise  the 
code.  Until  such  time  that  this  actual  data  is  available, 
simulated  versions  of  the  data  must  be  used  for  testing  purposes. 

6.  NOTES.  Not  Applicable. 
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APPENDIX  E 
MASTER  TEST  PLAN 

SYSTEM  LEVEL  TEST  AND  EVALUATION  (SLT&E)  PLANS 
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The  Air  Force  components,  locations,  Air  Force  Specialties 
( AFSs) ,  and  organizations  that  will  participate  in  the  SLT&E  for 
the  AOTS  are  as  follows: 

1.  Active  Air  Force,  Bergstrom  AFB,  TX 

AFS  426X2  -  67  CRS  and  67  AGS 

AFS  431X1  -  67  AGS  and  67  EMS 

AFS  732X0  -  67  CSG  and  67  SPS 

AFS  811X0  -  67  SPS 

AFS  811X2  -  67  SPS 

2.  Air  Force  Reserves,  Bergstrom  AFB,  TX 
AFS  426X2  -  924  CAMS 

AFS  431X1  -  924  CAMS 
AFS  732X0  -  924  TFG 
AFS  811X0  -  924  WSF 

3.  Air  National  Guard,  Ellington  ANGB,  TX 
AFS  426X2  -  147  CAMS 

AFS  431X1  -  147  CAMS 
AFS  732X0  -  147  CAMS/MSS 
AFS  811X0  -  147  SPF 
AFS  811X2  -  147  SPF 

The  following  figure  reflects  for  each  Air  Force  component  the 
numbers  of  personnel  within  each  AFS  and  workcenter  type  that  are 
to  be  involved  in  the  SLT&E  for  the  AOTS. 


AFS 

WORKCENTER 

NUMBER  ASSIGNED  1 

TOTALS 

ACTIUE 

RESERUE  I  GUAR0  j 

PERSONNEL 

732X0 

0PI1Q 

OR 

13 

1 

2 

1 

2 

1  19 

! 

■Jhi  ENGINE  MAIN! 
426X2 

JEIH  1  25 

AMU  9 

26 

32 

T 

|  92 

i 

AIRCRAFT  MAINT. 
431X1 

A  HU  li  44  i- 

PHASE  '  j  24  T 

73 

21 

66 

i 

j  228 

HnMI 

FLIGHTS 

SPA 

l 

j  67 

!  w 

43 

66 

!  176 

(  H 

j  197 

165 

167 

|  529 

j  197  165 

Ns  for  Control  Groups  will  b 
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SECTION _ PAGE 

INTRODUCTION  F-  3 

BI-WEEKLY  STATUS  UPDATE  REPORT  F-  4 


INTRODUCTION 


This  appendix  contains  a  Primavera  Bi-weekly  MTP  Status  Update, 
showing  the  status  of  each  MTP  activity  that  is  to  occur  during 
the  month  of  the  report. 

The  column  headings  in  the  report  reflect  the  following: 


ACTIVITY  ID  The  ID  number  assigned  to  the  activity. 


ORIG  DUR 

REM  DUR 


PCT 


CODE 


ACTIVITY 

DESCRIPTION 


Original  Duration.  Number  of  days  the  activity  was 
projected  to  require. 

Remaining  Duration.  Number  of  days  (from  date  of 
report)  that  are  estimated  to  be  required  to  com¬ 
plete  the  activity.  (If,  at  the  time  of  the  re¬ 
port,  it  has  been  determined  that  more  time  will  be 
required  than  originally  anticipated,  it  is  con¬ 
ceivable  that  the  number  of  days  remaining  could  be 
more  than  the  number  originally  assigned.) 

Percent  of  the  total  activity  that  has  been  com¬ 
pleted.  (In  the  sample  report,  none  has  been  com¬ 
pleted  because  the  projection  is  a  beginning  one.) 

Column  for  the  individual  responsible  for  carrying 
out  the  activity  to  initial  when  the  activity  has 
been  completed. 


Self-explanatory . 
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BALL  SYSTEMS  ENGINEERING  DIVISION 

PRIMAVERA  PROJECT  PLANNER 

REPORT  OATE  11FEB88  RUN  NO. 

MTP  BI-WEEKLY.  STATUS.  UPOATE 

49 

-  SLT&E  OATA 

ACQUISITION  PRO 

ACTIVITY  ORIG  REM 

ACTIVITY 

DESCRIPTION 

ID  OUR  DUR  PCT 

CODE  ‘ 

S  0  0  0  ANCHOR— S 

_  UPDATE  _ 


90S 

14 

14 

0 

UPDATE 

BAFB  DESIRE  PRODUCT  - 

IN 

92  S 

14 

14 

0 

. UPOATE 

EANGB  DESIRE  PRODUCT 

-  IN 

935 

5 

5 

0 

UPDATE 

EANGB  DESIRE  ENTERED 

INTO  DBMS 

1200 

9 

9* 

0 

UPOATE 

JANUARY  88  WITS  DATA 

1 20S 

— 

-• 

7 

V 

UPOATC 

ACTIVE  DUTY  WITS  1.277 

EPS /SHEETS  -  OUT 

1215 

/I 

4 

0 

UPOATC 

ACTIVE  DUTY  WITS  DATA 

SHEETS  -  IN 

1220 

n 

~> 

rt 

KJ 

UPOATE 

ACTIVE  DUTY  WITS  DATA 

INPUT 

1225 

3 

0 

AFRES  WITS  LETTERS/SHEETS  -  OUT 

UPOATE 


BALL  SYSTEMS  ENGINEERING  DIVISION 


PRIMAVERA  PROJECT  PLANNER 


REPORT  DATE  11FEB88  RUN  NO.  49  —  SLT&E  DATA  ACQUISITION  PRO 

MTP  BI-WEEKLY  STATUS  UPDATE 


ACTIVITY  ORIG  REM  ACTIVITY  DESCRIPTION 

10  OUR  DUR  PCT  CODE 


1235 

4 

4 

0 

UPDATE 

APRES  WITS  DATA  SHEETS  -  IN 

1240 

2 

2 

0 

UPDATE 

AFRES  WITS  DATA  INPUT 

124S 

J 

5 

'  0 

UPDATE 

AIR  GUARD  WITS  LETTERS/SHEETS  - 

OUT 

1 2S5 

4 

4 

0 

UPOATE 

AIR  GUARD  WITS  DATA  SHEETS  -  IN 

1260 

n 

im 

G 

UPOATE 

A  IP  GUARD  WITS  DATA  INPUT 

1265 

9 

9* 

0 

UPDATE 

PE8UARV  88  WITS  DATA 

a  t.  • 

J 

0 

UPOATC 

ACTIVE  OUT  WITS  LETTERS/SHEETS 

-  OUT 

1230 

4 

4 

0 

UPOATE 

ACTIVE  OUT V  WITS  DATA  SHEETS  - 

IN 

1285 

O 

c. 

2 

0 

ACTIVE  OUTY  WITS  DATA  INPUT 

UPOATE 


BALL  SYSTEMS  ENGINEERING  DIVISION  PRIMAVERA  PROJECT  PLANNER 

REPORT  DATE  11FEB88  RUN  NO.  49  —  SLT&E  OATA  ACQUISITION  PRO 

MTP  81-WEEKLY  STATUS  UPDATE 

_ }i 

ACTIVITY  ORIG  REM  ACTIVITY  DESCRIPTION 

ID  DUR  DUR  PCT  CODE 

1290  330  AFRES  WITS  LETTERS/SHEETS  -  OUT 

_  UPOATE  _ 

1300  4  40  AFRES  WITS  OATA  SHEETS  -  IN 

_  UPDATE  _ 

1305  220  AFRES  WITS  DATA  INPUT 

_  UPOATE  _ 

1310  3  3  C  AIR  GUARD  WITS  LETTERS/SHEETS  -  OUT 

_  UPDATE  _ 

1320  4  40  AIR  GUARD  WITS  DATA  SHEETS  -  IN 

_  UPDATE  _ 

1325  2  2  G  A • P  GUARD  WITS  OATA  INPUT 

_  UPDATE 


BALL  SYSTEMS  ENGINEERING  DIVISION 


PRIMAVERA  PROJECT  PLANNER 


REPORT  DATE  11FEB88  RUN  NO.  49 


—  SLT&E  DATA  ACQUISITION  PRO 


MTP  BI-WEEKLY  STATUS  UPDATE 


ACTIVITY  ORIG  REM 

ID  DUR  OUR  PCT  CODE 


ACTIVITY  DESCRIPTION 


sise  10  10  e 


MDC  REPORT  IN 


UPDATE 


>161  10  10  0 


480  REPORT  IN 


UPDATE 


5 1 62  19  10  0 


MILAP  REPORT  IN 


UPDATE 


UPDATE 


DCR  "FIRS  LOOK"  ANAL  .'SIS  --  REPEAT  MX 


4025  10  10  0 


RECEIVE  BASE  CRIME  AND  TRAFFIC  ANALYST 


UPDATE 


UPC-ATT 


UPDATE 


EXTRACT  DATA 


ACTIVE  OA  SUMMARY  •'EC.-C  17  LETTER 


S00S  10  !£  0 


RECEIVE  ACTIVE  OA  SUN ",  ARY  REPORT 


UPOATE 


S020  10  10  0 


RECEIVE  ACTIVE  QA  SUMMARY  REPORT 


UPOATE 
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BALL  SYSTEMS  ENGINEERING  DIVISION 


REPORT  OATE  11FEB88  RUN  NO.  49 
MTP  BI-WEEKLY  STATUS  UPOATE 


ACTIVITY  ORIG  REM 

JO  OUR  OUR  PCT 


sees 

Sj 

0 

UPDATE 

S05S 

10 

10 

0 

UPDATE 

5040 

■If 

Ct 

0 

UPOATE 

10 

10 

0 

— 

— 

_  UPDATE 

5055 

7 

0 

_  UPDATE 

i 


PRIMAUERA  PROJECT  PLANNER 
---  SLT&E  DATA  ACQUISITION  PRO 


ACTIVITY  DESCRIPTION 

EXTRACT  ACTIVE  OA  SUMMARY  REPORT  DATA 

RECEIVE  ACTIVE  OA  SUMMARY  REPORT 

EXTRACT  ACTIVE  OA  SUMMARY  REPORT  DATA 

RECEIVE  ACTIVE  OA  SUMMARY  REPORT 

EXTRACT  ACTIVE  OA  SUMMARY  REPORT  DATA 
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INTRODUCTION 


This  appendix  contains  sample  schedules  for  events  to  occur  in 
support  of  the  System  Level  Test  and  Evaluation  (SLT&E)  for  the 
Advanced  On-the-job  Training  System  (AOTS) .  Included  are: 

1.  Action  required  to  prepare  for  the  SLT&E. 

2.  Time  table  for  collection  of  SLT&E  data  after  the 
prototype  AOTS  has  been  implemented. 
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APPENDIX  H 


BASELINE  DATA  COLLECTION  PROCEDURES 

This  Appendix  describes  the  detailed  procedures  to  be  fol¬ 
lowed  in  acquiring  the  data  to  define  the  present  state  of  Air 
Force  On-the-job  Training  (ie.,  the  "baseline")  within  the  test 
workcenters  at  Bergstrom  AFB  and  Ellington  ANGB.  The  step-by- 
step  procedures  to  acquire  baseline  data  in  the  following  areas, 
as  well  as  Appendix  E  references,  are  included  in  this  Appendix, 
as  follows: 

H.l  Test  Subject  Demographic  Data . 2 

H. 2  OJT  records  (E-7A  &  B) . 3 

—  AF  Form  623  Folders 
—  Job  Qualification  Standards  (JQS) 

H. 3  Quality  Assurance  Summaries  (E-8A)  .  .  8 

—  67  TRW  Quality  Assurance  Quarterly  Summary 

—  924  TFG  Quality  Assurance  Quarterly  Summary 
—  Ellington  ANGB  Quality  Assurance  Data 

H.4  Security  Police  Quality  Control  Trend  (E-8A)  .  25 

—  Quarterly  Quality  Control  Trend  Analysis 

H . 5  Weekly  Inventory  of  Time  Spent  (WITS)  Sheets 

(E-11A,  15A,  23B) . 28 

H.6  Base  Crime  Statistics  (E-10A)  .  39 

—  Base  Crime  Analysis  Report 

H . 7  Aircraft  Maintenance  Data  (E-9A)  .  43 

—  MDC  Report 
—  MILAP  480  Report 

H. 8  Baseline  Surveys  (E-20A)  .  55 
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H.l  TEST  SUBJECT  DEMOGRAPHIC  DATA 


H.1.1  Data  To  Be  Collected.  Test  Subject  Demographic  Data  con¬ 
sists  of  specific  demographic  data  relating  to  each  test  subject 
and  will  be  obtained  from  the  AOTS  PDS  table. 

H.l. 2  Requirements  For  Data  Collection  Representative  ( DCR) . 
The  DCR  who  will  collect  the  data  via  the  AOTS  PDS  table  should 
be  an  1ST  SME  from  the  732X0  AFSC. 

H.l. 3  Frequency  Of  Data  Collection.  Data  collected  via  the  AOTS 
PDS  table  will  be  collected  once  on  each  subject  during  the 
baseline  data  acquisition  period. 

H.l. 4  Data  Collection  Procedure.  In  order  to  obtain  demographic 
data,  the  DCR  will  extract  the  applicable  data  from  the  AOTS  PDS 
table. 


H.1.5  Time  Required  For  Data  Collection.  Following  are  the  time 
requirements  for  requesting  and  receiving  the  demographic  data 
via  the  AOTS  PDS  table: 

0.12  mo  Data  input 

H.l. 6  Disposition  Of  Data.  When  the  DESIRE  data  is  received, 
the  DCR  will  deliver  the  product  to  the  Data  Input  Representative 
(DIR)  .  The  DIR  will  have  the  data  entered  into  the  DESIRE  data 
base. 
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H . 2  OJT  RECORD  DATA 


H.2.1  Data  To  Be  Collected.  Data  relevant  to  trainees7  progress 
and  qualification  on  tasks  that  make  up  their  duty  positions  will 
be  collected  as  specified  herein.  These  data  are  contained  in 
Air  Force  Forms  (AFFMS)  623,  On-the-Job  Training  Rec.or.ds.  „  AFFMS 
623  are  folders  that  contain  various  other  documents,  the  most 
significant  for  our  purposes  being  the  Job  Qualification  Standard 
(JQS) .  The  JQS  is  used  to  record  training  and  qualification 
status.  These  records  are  maintained  by  trainees7  immediate  su¬ 
pervisors  within  operational  workcenters.  Related  illustrations 
are  as  follows: 

OJT  Data  Recording  Form,  Figure — H.2-1,  p.  6 

Specialty  Training  Standard  (STS) — Figure  H.2-2,  p.  7 

II.  2. 2  Requirements  For  DCR.  The  training  record  data  collec¬ 
tion  will  require  the  expertise  of  an  OJT  training  specialist 
(751X1) .  The  only  additional  training  required  for  the  DCR  will 
be  a  familiarization  with  the  specific  procedures  for  the  data 
collection. 

H.2.3  Frequency  of  Data  Collection.  Data  collection  of  the  OJT 
record  will  be  performed  twice  during  Baseline:  at  the  beginning 
and  end  of  the  Baseline  period. 

H.2.4  Data  Collection  Procedures.  The  DCR  will  generate  a  list 
of  all  trainees  and  their  data  base  identification  numbers  by  use 
of  AOTS  PDS  table.  The  DCR  will  use  this  list  and  OJT  record 
data  sources  to  perform  the  data  collection  procedures  described 
below. 


H.2.4.1  Data  Locations.  The  DCR  will  be  provided  access  to  the 
training  records  at  each  office  by  the  point  of  contact  (PQC) . 
Refer  to  Attachments  2,  3,  and  4  at  the  end  of  Appendix  H  for 
POCs  and  addresses  for  obtaining  training  records. 


H.2.4. 2  Initial  Collection.  The  procedures  used  to  record 
OJT  task  information  are  the  same  for  all  Air  Force  components 

( I  .  p  . 
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Specialties  (i.e.  426XX,  431XX,  732XX,  and  811XX) .  The  DCR  will: 


-  Obtain  sufficient  number  of  copies  of  OJT  Data  Recording 
Forms  for  trainees  involved,  plus  extras  for  'the  purpose 
of  adding  new  names  as  required.  . 
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-  Coordinate  access  to  the  workcenter  with  the  respective 
supervisor. 

-  Obtain  from  the  workcenter  supervisor  the  training  fold¬ 
ers  (AF  Form  623)  for  all  OJT  trainees  in  the  workcenter. 

-  Fill  out  the  name  of  the  COLLECTOR  and  circle  the  COMPO¬ 
NENT  on  the  OJT  DATA  RECORDING  FORM. 

-  Referring  to  the  training  folder,  fill  out  the  NAME  of 
the  individual  and  his/her  SSAN. 

-  Enter  the  date  the  individual  was  assigned  to  the  duty 
position.  (Obtain  this  information  from  the  individual 
or  from  the  supervisor.) 

-  Locate  the  individual's  JQS  in  the  AF  Form  623.  The  JQS 
consists  of  the  STS/AF  Form  797.  The  JQS  is  usually  the 
top  document  on  the  right-hand  side  of  the  training 
folder. 

-  Enter  the  number  of  tasks  for  the  individual's  duty  posi¬ 
tion  in  the  TOTAL  #  TASKS  CIRCLED  DURING  INITIAL  COLLEC¬ 
TION  block  on  the  Data  Recording  Form.  Count  the  number 
of  circled  tasks  on  the  STS/AF  Form  797  to  determine  the 
number  to  list. 

-  Enter  the  TOTAL  #  TASKS  OPEN  in  the  appropriate  block. 

-  If  individual  is  position  qualified,  enter  the  date  the 
last  open  task  was  closed  in  the  DATE  POSITION  QUALIFIED 
block  of  the  heading  of  the  OJT  Data  Recording  Form. 

H.2.4.3  Final  Collection.  For  the  final  data  collection,  the 
DCR  will  take  with  him/her  the  Data  Recording  Forms  that  were 
filled  out  during  the  initial  collection. 

-  The  DCR  will  take  a  supply  of  blank  Data  Recording  forms 
to  the  workcenter  so  that  forms  can  be  filled  out  for  any 
new  personnel  assigned  to  the  workcenter. 

-  Refer  to  the  information  in  the  AF  Form  623  to  obtain 
necessary  information  for  filling  out  the  Data  Recording 
Form.  (For  any  new  personnel,  the  DCR  will  follow  the 
procedure  listed  above  for  initial  collection) . 

-  Count  and  record  the  total  number  of  tasks  circled  for 
the  FINAL  COLLECTION  in  an  individual's  STS/AF  Form  797. 


-  Count  and  record,  by  month,  the  TOTAL  #  TASKS  OPEN  as 
shown  on  the  individual's  STS/AF  Form  797. 

Count  and  record,  by  month,  the  TOTAL  #  TASKS  CLOSED  as 
shown  on  the  individual's  STS/AF  Form  797. 

If  individual  is  position  qualified,  enter  the  date  of 
qualification  in  the  designated  block  of  the  heading  of 
the  OJT  Data  Recording  Form. 

H.2.5  Time  Required  for  Data  Collection.  The  estimated  time 
requirements  for  the  data  collection  differ  for  the  initial  col¬ 
lection  and  the  final  collection. 


H.2.5. 1  Initial  Data  Collection  Time  Requirements:  (man  hours 

per  OCR) . 


OJT  records  collection  (per  work  center) 


0.25 

0.25 

12.25  ave. 
1.00  ave. 


Arrange  data  collection  visits 
Miscellaneous  travel  on-base 
Initial  extract  of  task  data 
Input  data 


H.2.5. 2  Final  Data  Collection  Time  Requirements. 


OJT  records  Collection  (per  work  center) 


0.25 

0.25 

24.50  ave. 
1.00  ave. 


Arrange  data  collection  visits 
Miscellaneous  travel  on-base 
Extract  data 
Input  data 


H.2.6  Disposition  of  Data.  The  DCR  will  deliver  the  Data 
Recording  Forms  to  the  DIR;  after  the  data  has  been  input  (except 
for  the  final  time) ,  the  DCR  will  obtain,  from  the  data  input 
person,  the  forms  for  use  in  recording  information  during  the 
next  data  collection  procedure. 


OJT  DATA  RECORDING  FORM 

NAME: _ 

SSAN: _ 

WORKCENTER: _ 

DATE  ASSIGNED  TO  WORKCENTER: _ 

DATE  POSITION  QUALIFIED: _ 

TOTAL  #  TASKS  CIRCLED  DURING  INITIAL  COLLECTION: 
TOTAL  #  TASKS  CIRCLED  DURING  MID  COLLECTION:, 
TOTAL  #  TASKS  CIRCLED  DURING  FINAL  COLLECTION: 


TOTAL  #  TASKS  OPEN  by  month: 

Oct  87 _ 

Nov  87 _ 

Dec  87 _ 

Jan  88  _ 

Feb  88 _ 

Mar  88 _ 

Apr  88  _ 

May  88 

r  - 

Jun  88  _ 

Jul  88 


TOTAL  #  TASKS  CLOSED  by  month: 

Oct  87 _ 

Nov  87 _ 

Dec  87 _ 

Jan  88  _ 

Feb  88 _ 

Mar  88 _ 

Apr  88  _ 

May  88 _ 

Jun  88 _ 

Jul  88 


COMPONENT:  Active  Reserve  ANG 


COLLECTOR: 


PRIVACY  ACT  STATEMENT 

„„„ _ _ _ AUTHORITY:  44  USC  8101;  10  USC  8012  and  EO  0307 

PRINCIPAL  PURPOSE:  The  SSAN  will  be  used  to  identify  personnel  participating  in  the 
DMiitvm  ,  _  .  „  Advanced  On-The-Job  Training  System  Prototype, 

nicmmrml  to  1 a  wr\ 5 ^*, current  OJT  program.  2.  Determine  the  effectiveness  of  the  AOTS prototype. 

•  i  j  j  •  ivv?  Am2*A”PATORY;  The  SSAN  is  mandatory  to  make  positive  identification  of  individuals  in  relation  to  other  personnel 
included  in  the  AOTS  prototype.  Failure  to  provide  thie  information  will  prevent  monitoring  of  training  related  data,  thereby  jeopardising 

_ the  ability  of  AOTS  to  perform  its  mission. 


Figure  H.2-1 
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TASKS.  KNOWLEDGE  AND 
TECHNICAL  REFERENCES 


CERTIFICATION  FOR  OJT 


SUM 

Oat* 


Con*3Utlon 

Oat* 


C«Mllylnt 

OMIclat'i 

Initial* 


Tnlnit’i 

Inltiait 


3.  PROFICIENCY  COOES  USED  TO  INDI¬ 
CATE  TRAINING/INFORMATION  PROVIOEO 


3  Si ftT  L 


O)  (2) 

Count  I  COC 


S  Skill  L«v«i 


l«>  <2> 

Court*  I  COC 
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*13.  UTILITY  SYSTEMS  (continued) 

(2)  Dram 

(3)  Dchumidifiers 

(a)  Inspect 

(b)  Remove 

(c)  install 


Q 


Remove  LOX  converters 
^gT^  Install  t.OX  converters 
h.  Inspect 


(1) 

Overheat  warning  system 

<© 

Fire  detection  system 

(3) 

rire  extinguishing  system 

(4) 

Air  conditioning  system 

© 

Oxygen  system 

(6) 

Pressurization  system 

(7) 

Rain  removal  system 

(8) 

Windshield  wiper  system 

Bleed  air  system 

(10) 

Anti -icing  system 

i.  Troubleshoot 

(1) 

Overheat  warning  system 

(2) 

Fire  detection  system 

(3) 

Fire  extinguishing  system 

(4) 

Air  conditioning  system 

(b) 

Oxygen  system 

(6) 

Pressurization  system 

(7) 

Rain  removal  system 

(8) 

Windshield  wiper  system 

(9) 

Bleed  air  system 

(10) 

Anti-icing  system 

14.  FLIGHT  CONTROL  SYSTEMS 
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Figure  H.2-2 


H-7 


H.  3  67  TRW  QUALITY  ASSURANCE  PROGRAM  QUARTERLY  SUMMARY,  924  TFG 
QUALITY  ASSURANCE  PROGRAM  QUARTERLY  SUMMARY,  AND  ELLINGTON 
QUALITY  ASSURANCE  DATA  COLLECTION  PROCEDURES 


H.3.1  Quality  Assurance  Program  Quarterly  Summary . 

H.3.1.1  Data  to  be  Collected.  The  67  TRW  Quality  Assurance  (QA) 
Program  Quarterly  Summary  will  be  sent  directly  to  AFHRL.  The 
active  components  use  the  summary  to  rate  the  qua? ity  of  the 
aircraft  maintenance,  propulsion,  and  phase  inspection  workcen- 
ters.  The  DCR  will  be  interested  in  three  of  the  sections  of  the 
Quarterly  Summary. 

-  Section  II,  67th  Aircraft  Generation  Squadron;  the  sub¬ 
section  of  interest  is  titled  "12  Aircraft  Maintenance 
Unit." 

-  Section  III,  67th  Component  Repair  Squadron;  the  subsec¬ 
tion  of  interest  is  the  one  titled  "Propulsion  Branch." 

-  Section  IV,  67th  Equipment  Maintenance  Squadron;  the 
subsection  of  interest  is  the  one  titled  "Maintenance 
Branch. " 

H.3.1. 2  Requirements  for  DCR.  The  QA  data  collection  personnel 
for  Bergstrom  AFB  do  not  need  a  particular  background  experience 
but  will  require  some  training  in  the  procedures  for  extracting 
data  from  the  monthly  summaries. 

H.3.1. 3  Frequency  of  Data  Collection.  Quarterly. 

H.3.1. 4  Data  Collection  Procedure.  POC  is  MSgt  Hendrickson, 
MAQ,  building  1603.  AFHRL  will  receive,  quarterly,  the 

QA  Quarterly  Summary — Figure  H.3-1,  p.  13. 

The  DCR  will  turn  to  Section  II  -  67th  Aircraft  Generation 
Squadron,  look  under  the  subsection  "12  AMU,"  and  find  the  areas 
"Technical"  and  "Personnel." 

(See  Figure  H.3~la,  p.  14.) 

Enter  the  totals  from  the  technical  inspections  (E,  S,  U, 
TOTAL)  onto  the : 

QUALITY  ASSURANCE  PROGRAM  MONTHLY  SUMMARY 
Data  Recording  Form-Figure  H.3-2,  p.  17 
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along  the  row  titled  "AMU".  Ensure  totals  are  recorded 
under  the  corresponding  column  headings  of  the  Data  Re¬ 
cording  Form. 

-  Enter  the  totals  from  the  “Personnel"  section  of  the  QA 
Program  Quarterly  Summary  onto  the  QA  Recording  Form 
along  the  row  titled  "AMU"  under  the  corresponding  column 
headings.  The  numbers  from  the  QA  Program  Quarterly 
Summary  (Figure  H.3~la,  p.  14)  are  circled  to  demonstrate 
which  numbers  to  extract  for  the  Baseline  data  collec¬ 
tion. 

-  Jet  Engine  Intermediate  Maintenance  (JEIM) :  Data  for  the 
JEIM  shop  is  contained  in  the  report  entitled  "Propulsion 
Branch. " 

-  Enter  the  totals  for  the  section  as  circled  on  the  exam¬ 
ple  in 


Figure  H.3-lb,  p.  15. 

-  Write  these  sums  on  the  "Propulsion"  row  of  the  Data  Re¬ 
cording  Form  under  the  corresponding  column. 

-  Enter  the  numbers  along  the  "TOTAL"  row  of  the 

QA  Program  Quarterly  Summary — Figure  H.3-lc,  p.  16 

onto  the  Data  Recording  Form  along  the  row  titled  "Main¬ 
tenance."  (Figure  H.3-2,  p.  17) 

-  Ensure  totals  are  recorded  under  the  corresponding  column 
headings  of  the  Data  Recording  Form. 

H.3.1.5  Time  Required  for  Data  Collection. 

QA  Quarterly  Summary  Collection  (Bergstrom  AFB) 

0.33  1ST  SME  looks  over  summary 
0.75  DCR  extract  data  from  summary 

0.13  Input  data  (Includes  Reserves) 

H*3*2  924. TFG  Quality  Assurance  Program  Quarterly  Summary. 

H.3.2.1  Data  to  be  Collected.  The  924  TFG  QA  Quarterly  Summary 
is  used  by  the  Reserve  component  to  rate  the  quality  of  aircraft 
maintenance,  propulsion,  and  inspection  workcenters.  This  report 
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is  significantly  different  from  the  67  TRW  QA  Quarterly  Summary 
(active  duty) . 

H.3.2.2  Requirements  for  the  DCR.  The  requirements  are  the  same 
as  for  the  QA  Quarterly  Summary  in  Paragraph  H.3ol.2. 

H.3.2.3  Frequency  of  Data  Collection.  Quarterly. 

H.3.2.4  Data  Collection  Procedure.  Included  is  an  example  of 
the  924th  TFG 

Reserve  Unit  QA  Quarterly  Report — Figure  H.3-3,  p.  18. 

-  Add  the  numbers  for  the  following  types  of  inspections: 
Pre  Flight,  Launch/EOR,  25  Hr  Flight  Line  Inspection, 
Thru  Flight/BPO,  and  Engine  Bay  Activities  (boxed  items) . 

(See  Figure  H.3-3a,  p.  19.) 

Record  these  sums  on  the  Data  Recording  Form  along  the 
row  titled  "AMU'1  under  the  section  of  the  form  for  the 
"Reserve"  component  (Figure  H.3-2,  p.  17). 

-  Enter  the  numbers  from  the  engine  installation  line  as 
before. 

The  Reserve  Quality  Assurance  Office  aggregates  the  data  col¬ 
lected  during  personnel  evaluations.  Therefore,  such  data  must 
be  collected  in  aggregate  form.  The  data  collector  must  locate  a 
chart  in  the  Quarterly  Report  entitled: 

PERSONNEL  EVALUATION  RATING— Figure  H.3-3b,  p.  20. 

The  information  in  the  row  labeled  "TOTAL"  is  entered  on'  the  Data 

Recording  Form  on  the  row  labeled  "Reserves"  in  the  "TOTAL  PASS 
FAIL"  boxes.  '  ' 


An  example  of  the  Quality  Evaluation  Inspection  (QVI)  for  propul- 

>se<r  the  quarterly  charts  labeled  TYPE  INSPECTION  BY  EQUIP¬ 
MENT  is  included  in:  '  - 


Figure  H.3-3c  p.  21. 
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H.3.2.5 


Time  Required  for  Data  Collection. 


QA  Quarterly  Summary  Collection  (Bergstrom  Reserves) 

0.33  1ST  SME  looks  over  summary 

1.00  Extract  data  from  summary 

H.3.3  Ellington  ANGB  Quality  Assurance  Data. 

H.3.3.1  Data  to  be  Collected.  The  DCR  will  collect  QA  data  from 
the  QA  files  at  Ellington  ANGB. 

H .  3 . 3 . 2  Requirements  for  DCR.  The  DCR  for  the  Ellington  QA 
data  should  be  an  1ST  SME  (426X2,  431x1)  or  someone  trained  by 

an  expert  on  the  collection  of  this  raw  QA  data. 

H .  3 . 3 . 3  Frequency  of  Data  Collection .  Monthly . 

H-3-3*4  Data  Collection  Procedure.  The  quality  assurance  data 
from  the  Air  National  Guard  are  collected  from: 

Air  Force  Forms  2419 — Figure  H.3-4,  p.  22. 

These  forms  are  located  in  the  QA  Office  at  Ellington  ANGB.  The 
is  located  in  Building  1382,  and  the  points  of  contact  are 
either  CMSgt  McCoubrey  or  Betty  Holt.  The  data  from  the  AF  Forms 
2419  is  processed  as  follows: 

H.3.3. 4.1  Enter  the  QA  data  on  the  Data  Recording  Form  (Figure 
H.3-2,  p.  17)  . 

-  A  431X1  SME  will  determine  which  AF  Forms  2419  will  be 
analyzed. 

-  First,  the  workcenter  being  evaluated  is  identified. 
This  is  done  by  an  examination  of  the  (SHOP/BRANCH)  iden¬ 
tified  in  Block  II  of  the  AF  Form  2419.  For  Baseline 
data  collection  purposes,  tne  shops  from  which  data  will 
be  collected  are  the  Propulsion  and  the  AMU. 

-  If  the  name  appears  in  Section  III,  the  AF  2419  was  used 
in  a  personnel  evaluation. 

If  the  AF  Form  2419  was  used  for  a  personnel  evaluation: 

Figure  H.3-5,  p.  23  (Block  III), 
record  the  "rating"  indicated  in  block  III  on  the  Data 
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Recording  Form  under  the  appropriate  heading  of  either 
Satisfactory  (SAT)  or  Unsatisfactory  (UN) . 

-  If  there  is  no  entry  in  block  III,  this  is  considered  a 
technical  evaluation;  and  an  entry  should  appear  in 
Block  IV: 

Figure  H.3-6,  p.  24. 

-  Record  all  other  evaluations  on  the  Data  Recording  Form 
as  follows: 

—  Block  V  "RATING”  provides  the  quality  inspection  rat¬ 
ing.  Indicate  the  rating  on  the  Data  Recording  Form 

under  the  columns  labeled  "Technical  Evaluations" 
(excellent,  satisfactory,  unsatisfactory) . 

—  When  all  the  AF  Forms  2419  prepared  during  the  cur¬ 
rent  month  have  been  reviewed  and  the  data  has  been 

recorded  in  the  columns  provided  on  the  Data  Record¬ 
ing  Form,  the  DCR  must  sum  each  column  on  the  Data 
Recording  Form. 

H.3.3.5  Time  Required  for  Data  Collection. 

QA  Data  Collection  (Ellington  ANGB) 

0.25  Arrange  visit  to  workcenter 

0.25  Miscellaneous  on-base  travel 

5.00  Extract  data  from  AF  Form  2419 

0 . 10  Input  data 

H.3.5  Disposition  of  Data.  Once  the  Data  Recording  Forms  have 
been  completed  and  summed  per  the  procedures  above,  the  informa¬ 
tion  is  provided  to  the  DIR.  The  DIR  will  have  the  totals  con¬ 
tained  on  the  Data  Recording  Form  entered  into  the  appropriate  QA 
data  table. 
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Figure  H.3-lb 
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QUALITY  ASSURANCE 


>.  FOOTE,  SMSgt,  USAFR 
Assurance  Supervisor 


MONTHLY  MAINTENANCE  INSPECTIONS  DAT5UL  .  SEP  1987 


ROUTING  AND  REVIEW  OF  QUALITY  CONTROL  REPORTS 


«CPO»T  WOUTINC  SCOUCNCC 


OIMLITV  COWTHOL  INSPECTION  (VALUATION  SuMUAPY 


Overall  condition  of  the  aircraft  was  good,  except  for  the  1  major  discrepancy, 
F.O.  in  the  cockpit.  The  last  5  BPO  QVI  reports  for  this  AMU  were  reviewed  and 
each  inspection  had  4-7  missed  carded  items.  A  trend  appears  to  be  developing 
in  this  type  inspection.  Available  PR/BPO  cards  were  checked  and  found 
satisfactory.  ' 


ppintcOnamc  AN  o  CP  A  or  or  inspcc  roiTf  t 

j 15  Aug  82  J.  THOMPSON  MSGT 
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Sample  AF  Form  2419,  Routing  and  Review  of  Quality  Control  Reports 


'igure  H.3-4 
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ROUTING  AND  REVIEW  OF  QUALITY  CONTROL  REPORTS 
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QUALITY  CONTROL  INSPECTION  EVALUATION  SUMMARY 


Overall  condition  of  the  aircraft  was  good,  except  for  the  1  major  discrepancy, 
F.O.  In  the  cockpit.  The  last  5  BPO  QVI  reports  for  this  AMU  were  reviewed  and 
each  inspection  had  4-7  missed  carded  Items.  A  trend  appears  to  be  developing 
in  this  type  Inspection.  Available  PR/BPO  cards  were  checked  and  found 
satisfactory.  '*  "  • 
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Figure  H.3-5 

Sample  AF  Form  2419,  Routing  and  Review  of  Quality  Control  Reports 
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ROUTING  AND  REVIEW  OF  QUAUTY  CONTROL  REPORTS 


PCPOPT  POUTING  SCOUCNCC 


jufftNir. 

Tfi  Ao?  1ST 


MAAAC 


MAAA 


18  Aug  8?~ 


_MAA_ 

HAO 


19  Aug  82 


i 


20  Aug  82 


MA 


23  Aug  82 


JiAJL 


24  Aug  82 


CrNCRAl  IN'OPMATION 


AGS 


I  OUl nwfN  1 

F-1 5C 


KMUL  MUMIC.M 

79-0091 


OA  T  C 

15Auq82 


PCPSONNCL  EVACUATION  PATINO 


N»«f  »(,•«!,  /  m»I,  Kirfrfl*  InttUI) 


T ¥*f  C  VAC 


T  V  PC  C  VALUATION  ‘INSPECTION  PC«*OPMCO 


PCPSONNCi.  CPPORS 


P'< 


V. 

EQUIPMENT  INSPECTION  PAYING 

C  A  A  O  • Y IMS 

Oi»C"C»»NCr  j 

r'.oclNtACC 

e  Aitcirat 

n»i  >nc 

CVAlOATtO 

OCCUCTION 

6 

OISCPC^AHClCS  ev  CATEGORY 


TS7 

JjLL. 


maj 

0 


MIN 

u 


0 


M,N 

0 


MA  J 
0 


MIN 

0 


MAJ 

0 


MIN 

0 


OU*Utr  CONTROL  INSPECTION.  EVALUATION  SUMMARY 


Overall  condition  of  the  aircraft  was  good,  except  for  the  1  major  discrepancy, 
F.O.  in  the  cockpit.  The  last  5  BPO  QVI  reports  for  this  AMI)  were  reviewed  and 
each  inspection  had  4-7  missed  carded  items.  A  trend  appears  to  be  developing 
in  this  type  Inspection.  Available  PR/BPO  cards  were  checked  and  found 
satisfactory.  '  *  * ' 
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Figure  H.3-6. 

Sample  AF  Form  2419,  Routing  and  Review  of  Quality  Control  Reports 
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H.  4 


SECURITY  POLICE  QUALITY  CONTROL  TREND  PROCEDURES 


H.4.1  Data  to  be  Collected.  AFFRL  will  receive,  each  quarter 
the: 

Security  Police  Quality  Control  Trend  Analysis 
(Figure  H.4-1,  p.  26) 

Data  collected  will  be  the  average  quarterly  QC  Evaluation  scores 
of  SPA,  SPO,  SPOL-A,  SPOL-B,  SPOL-C,  and  SPOL-D. 

H.4.2  Requirement  for  DCR.  The  DCRs  for  Security  Police  Quality 
Control  Trend  Analysis  reports  do  not  require  specialized  ex¬ 
perience  or  training. 

H.4.3  Frequency  of  Data  Collection.  Security  Police  Quality 
Control  Analysis  data  will  be  collected  quarterly. 

H.4.4  Data  Collection  Procedure.  AFHRL  will  receive  the 
Security  Police  Quality  Control  Trend  Analysis  quarterly.  The 
DCR  will  record  information  on  the: 

Security  Police  Quality  Control  Data  Recording  Form 
(Figure  H.4-2,  p.  27) 

The  following  information  will  be  recorded  on  the  Recording  Form 
in  the  appropriate  columns  and  blank,  as  labeled: 

1)  Quarter. 

2 )  Date . 

3)  Average  score  of  each  flight. 

4)  DCR' s  name. 

H.4.5  Time  Requirements.  Time  requirements  for  the  collection 
of  Security  Police  Quality  Control  Trend  Analysis  data  are  as 
follows: 

.07  hr.  Record  information  on  Data  Recording  Form 
.07  hr.  Input  data 

H*4.6  Disposition  of  Data.  The  DCR  will  deliver  the  completed 
Security  Police  Quality  Control  Data  Recording  Forms  to  the  DIR. 
The  DIR  will  then  have  the  aara  entered  in  the  Security  Police 
Quality  Control  Data  Table. 
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soever  v-j  Quarter  Quality  Control  Trend  Analysis 


HI  TURN 


1.  During  this  quarter  the  following,  job  positions  were  notrd  as 
requiring  attention  or  additional  training. 

a.  ART/SRT-  Handcuffing  procedures  (repeat),  .and  combat 
f  ormati-on. 


b.  Flight  Chief  -  Detention  Facility  Operations,  Custom 
Procedures . 

c.  L/E  Patrol  -  Use  of  speed  measuring  devices  (repeat)  and 
prisoner  custody  and  control, 

2.  The  following  statistical  data  indicates  the  number  of  QCE's 
conducted  this  quarter. 


POSITIONS 


TQTAL 


a.  Fit  Chief  033  6 

b.  Art/Sr t  1  2  2  5 

c.  Desk  Sgt  012  3 

d.  LE  Patrol  0  1  10  11 

e.  Armory  003  3 

f.  Gate  Guard  0  2  1  3 

g.  Dual  Cert.  0  2  4  6 


I  MAs 

a.  Fit  Chief  100  1 

b.  Art/Srt  J__  j2_  _1_ 

3  U  25  39 

Failures:  1 

Failure  Rate:  2.5 


3.  The,  following  shows  a  ranking  of  sections  and  flights 
according  to  overall  QCE  scores. 

SPA:  90.  IX  SPOL-A:  No  T^tir.g  SPOL-C:  86.8? 

SP0:  89. OS  SPOL-B:  39.0?  ^  3P0L-D:  85. 3X 

/’i  out  caion 
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Figure  H.4-1 


Security  Police  Quality  Control 

Trend  Data  Recording  Form 
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Figure  H.4-2 
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H. 5  WEEKLY  INVENTORY  OF  TIME  SPENT  (WITS)  SHEET  COLLECTION  PRO¬ 
CEDURES 


H.5.1  Data  To  Be  Collected.  The  WITS  Sheets  used  for  the 
baseline  data  collection  are  time  sheets  specifically  designed  by 
AFHRL  for  use  in  the  AOTS  baseline  and  SLT&E  data  gathering 
process.  WITS  sheets  track  the  number  of  hours  spent  on  a 
variety  of  specific  training  activities  by  various  test  subject 
personnel . 

H.5.2  Requirements  For  DCR.  It  is  not  necessary  for  the  DCR  to 
have  a  particular  experience  background.  The  DCR  must  be  capable 
of  following  instructions  and  answering  questions  in  the  workcen- 
ters  regarding  the  proper  method  for  filling  out  WITS  sheets. 
The  DCR  will  also  be  required  to  perform  simple  addition  on  the 
time  sheet. 

H.5.3  Frequency  of  Data  Collection.  WITS  sheets  will  be  filled 
out  and  collected  biweekly  for  all  Active  Duty  component  workcen- 
ters  at  Bergstrom  AFB.  Test  personnel  in  Active  Duty  workcenters 
will  fill  out  a  WITS  sheet  for  every  other  week  during  the  col¬ 
lection  period.  WITS  sheets  will  be  filled  out  and  collected 
biweekly  in  the  Reserve  workcenters  at  Bergstrom  AFB  and  the  Air 
National  Guard  workcenters  at  Ellington  ANGB.  Reserve  and  Air 
National  Guard  personnel  will  fill  out  WITS  sheets  only  on  UTA 
weekends  and  annual  tour. 

H.5.4  Data  Collection  Procedure. 

H.5.4. 1  Initial  Data  Collection  Procedure.  In  performing  the 
required  initial  WITS  sheet  distribution  and  collection  tasks, 
the  following  will  be  performed: 

THE  DCR  WILL 

-  Obtain  an  appropriate  number  of  WITS  sheets,  Instructions 
for  filling  them  out,  and  Data  Collection  Coordination 
Worksheets  to  distribute  to  the  field. 

-  Package  WITS  sheets,  Instructions,  and  Data  Collection 
Coordination  Worksheets  for  distribution  to  POCs. 

-  Call  a  Briefing  Meeting  of  all  POCs  for  Actives  and  Re¬ 
serves,  to  be  held  at  Bergstrom. 

-  Conduct  the  Briefing  Meeting,  explaining  data  to  be  col¬ 
lected  from  workcenters,  collection  dates,  details  of 
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filling  out  WITS  sheets,  and  responsibilities  of  POCs  in 
WITS  sheet  collection — including  accounting  for  all  per¬ 
sonnel  who  have  not  filled  out  a  WITS  sheet  (TDY,  trans¬ 
ferred,  etc) .  WITS  sheet  packages  will  be  handed  out  to 
focal  point  personnel;  DCR  must  be  prepared  to  answer 
questions  at  this  meeting. 

-  Travel  to  Ellington  ANGB  to  hold  another  Briefing  (same 
process  as  described  above) . 

THE  STAFF  AGENCY  FOCAL  POINT  AT  THE  UNIT  WILL 

Distribute  WITS  sheets  to  the  participants,  collect  them 
at  the  end  of  the  appropriate  week  or  time  period,  and 
fill  out  the  Data  Collection  Coordination  Worksheet  to 
account  for  any  WITS  sheets  that  are  missing. 

THE  DCR  WILL 

-  Go  to  each  staff  agency  focal  point  at  Bergstrom  to  pick 
up  completed  WITS  sheets  and  Coordination  Worksheets  at 
the  end  of  the  first  week  of  WITS  sheets7  being  filled 
out.  Ellington  WITS  sheets  and  Coordination  Worksheets 
will  be  mailed  to  DCR. 

Examine  the  Data  Collection  Coordination  Worksheets  to 
insure  that  all  WITS  have  either  been  collected  or  have 
been  listed  with  an  acceptable  reason  for  their  absence. 
If  there  is  a  discrepancy,  the  DCR  will  check  with  the 
staff  agency  focal  point  to  resolve  the  problem. 

Examine  the  face  of  the  WITS  sheets  to  insure  that  all 
blanks  have  been  filled  out,  as  appropriate. 

Perform  a  quick-look  analysis  to  insure  that  figures  are 
plausible. 

-  Deliver  the  WITS  sheets  to  DIR  for  input  to  the  appropri¬ 
ate  data  table. 


H.5.4.2  Routine  Data  Collection  Procedure.  In  performing  the 
required  routine  WITS  sheet  distribution  and  collection  respon¬ 
sibilities,  the  following  activities  will  be  accomplished; 
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THE  DCR  WILL 


-  Obtain  a  sufficient  number  of  WITS  sheets,  instructions 
for  filling  them  out,  and  Data  Collection  Coordination 
Worksheets  to  fulfill  requirements  and  make  distribution 
to  each  staff  agency  focal  point. 

STAFF  AGENCY  FOCAL  POINTS  WILL 

Distribute  WITS  sheets  to  the  participants,  collect  them 
at  the  end  of  the  appropriate  week  or  time  period,  fill 
out  the  Data  Collection  Coordination  Worksheet  to  account 
for  all  WITS  sheets  that  were  distributed,  and  return 
both  the  WITS  sheets  and  the  Coordination  Worksheet  to 
the  DCR. 

THE  DCR  WILL 

-  Examine  the  Data  Collection  Coordination  Worksheets  to 
insure  that  all  WITS  have  either  been  collected  or  have 
been  listed  with  an  acceptable  reason  for  their  absence. 
If  there  is  a  discrepancy,  the  DCR  will  check  with  the 
staff  agency  focal  point  to  resolve  the  problem. 

-  Examine  the  WITS  sheets  to  insure  that  the  data  appears 
feasible  by  checking  the  following: 

—  Duty  day  is  no  more  ■‘-han  18  hours. 

Time  allotments  within  a  given  category  are  not  iden¬ 
tical  throughout  the  week. 

H • 5 • 4 • 3  General  WITS  Sheet  Types  And  Delivery  Instructions. 
There  are  three  different  types  of  test  personnel  who  will  fill 
out  WITS  sheets.  In  order  to  accommodate  these  three  types  of 
personnel,  three  types  of  WITS  sheets  have  been  developed.  The 
types  of  WITS  sheets  and  the  Data  Coordination  Worksheet  used  in 
the  data  collection  activity  are: 

1)  Commander  WITS  Sheets— —Instructions  on  reverse  side 

(Figure  H.5-1,  p.  34) 

2)  Unit  Training  Manager  WITS  Sheets — Instructions  on 
reverse  side 

(Figure  H.5-2,  p.  35) 
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3)  Supervisor,  Evaluator,  Trainer,  and  Trainee  WITS 
Sheets 

(Figure  H.5-3,  p.  36) 


4)  Instructions — Supervisor,  Evaluator,  Trainer  WITS 
Sheets  (Trainee  instructions  on  reverse  side) 

(Figure  H.5-4,  p.  37) 

5)  Data  Coordination  Worksheet — Figure  H.5-5,  p.  38 

Commander  WITS  sheets  are  given  to  Squadron  Commanders;  Unit 
Training  Manager  WITS  sheets  are  given  to  the  base  OJT  Training 
Manager  and  the  Unit  Training  Managers  of  the  various  workcen- 
ters.  The  workcenter  personnel  will  receive  the  Supervisor, 
Evaluator,  Trainer,  and  Trainee  WITS  sheets  and  will  fill  out  the 
sections  which  apply  to  them. 

Following  are  specific  procedural  instructions  to  be  given 
to  test  subjects  for  filling  out  the  appropriate  type  of  WITS 
sheet: 


The  WITS  sheet  is  a  time  sheet  used  to  measure  the  amount  of 
time  that  you  typically  spend  on  training  or  related  activities. 
It  is  understood  that  in  many  cases  you  may  act  in  more  than  one 
of  the  capacities  listed  below  (for  example,  supervisor  and 
trainer)  .  If  this  is  the  case,  it  will  be  necessary  to  fill  in 
the  time  spent  in  each  of  the  roles  that  you  play  in  OJT.  If  you 
are  acting  in  more  than  one  role  at  a  time,  record  on  your  WITS 
sheet  the  time  spent  doing  tasks  principally  related  to  a  par¬ 
ticular  OJT  role.  In  order  to  fill  out  a  WITS  sheet,  enter  your 
name  and  the  date  of  the  last  day  of  the  applicable  workweek  or 
time  period. 


Trainees:  For  each  day  of  the  week  fill  out  blocks  for  Num¬ 
bers  1,  10,  11,  and  any  others  that  might  apply. 


Supervisors,  Evaluators,  or  Trainers:  For  each  day  of  the 
week  fill  out  the  blocks  for  Numbers  1  through  3  4  as  they  apply. 

Unit  Training  Manager:  For  each  day  of  the  week  fill  out 
the  blocks  for  Numbers  l  through  8  as  they  apply. 


Sauadron  Comma nrterq* 

—  •*  —  —  ~  —  -  *  wiia  y  u  x. 

bers  l  through  9  as  they  apply. 


-  - _ 1- 

WCStSJS. 


OUu  nuiu 
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For  WITS  Sheet  POCs,  refer  to  Attachments  1,  2,  3,  and  4  at 
the  end  of  Appendix  H.  For  participants  who  will  fill  out  WITS 
Sheets,  refer  to  the  Attachments  2,  3,  and  4,  except  for  ANG  and 
Base  OJT:  for  those,  include  Attachment  1. 


The  total  number  of  WITS  Sheets  required  for  the  Active  Duty 
Component  workcenters  at  Bergstrom  AFB  for  each  collection  ac¬ 
tivity  (every  two  weeks)  are: 

5  -  Commander  WITS  Sheets 

6  -  Training  Manager  WITS  Sheets 

172  -  Supervisor,  Evaluator,  Trainer,  and  Trainee  WITS  Sheets 

The  total  number  of  WITS  sheets  required  for  the  Reserve 
workcenters  at  Bergstrom  AFB  for  each  collection  activity  (once  a 
month)  are: 

4  -  Commander  WITS  Sheets 
4  -  Training  Manager  WITS  Sheets 
152  -  Supervisor,  Evaluator,  Trainer,  and  Trainee  WITS  Sheets 

The  total  number  of  WITS  sheets  required  for  Ellington  ANGB 
for  each  collection  activity  (once  a  month)  are: 

3  -  Commander  WITS  Sheets4  -  Training  Manager  WITS  Sheets 

4  -  Additional  Training  Manager  WITS  Sheets  during  UTA  weekends 
175  -  Supervisor,  Evaluator,  Trainer,  and  Trainee  WITS  Sheets 

H.5.5  Time  Required  For  Data  collection. 

Initial  Weekly  Inventory  of  Time  Spent  (WITS)  Sheet  Collec¬ 
tion  (For  One  Workcenter  At  Bergstrom) 

0.25  Arrange  meeting  at  workcenter 

0.16  Copy  and  prepare  WITS  sheets  for 

distribution 

0.25  Miscellaneous  travel  on-base 

0.75  Collect  and  distribute  WITS  sheets  in 

workcenter 


Weekly  Inventory  Of  Time  Spent  (WITS  sheet)  Collection 
(For  One  Workcenter  At  Ellington) 

0.25  Arrange  meeting  at  workcenter 

0.16  Copy  and  prepare  WITS  sheets  for 

distribution 
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0.25 

0.50 


Miscellaneous  travel  on-base 
Collect  and  distribute  WITS  sheets  in 
workcenter 


Routine  Weekly  Inventory  of  Time  Spent  (WITS)  Sheet  Collection 
(For  Any  Workcenter) 

0.16  Copy  and  prepare  WITS  sheets  for  mailing 

0.16  Mail  WITS  sheets 


Total  Data  Input  Time,  By  Type  WITS 

2.67  mo  Commander  WITS 

4.57  mo  Unit  Training  Mgr.  WITS 

18.81  mo  Supervisor/Evaluator/Trainer/Trainee  WITS 

5 . 6  Disposition  of  Data.  After  the  WITS  sheets  have  been  col¬ 
lected  and  the  above  actions  performed,  the  DCR  will  deliver  them 
to  the  DIR. 
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UNIT  COMMANDER  -ITS  SHEET  INSTRUCTIONS 


The  WITS  (Weekly  Inventory  of  Time  Spent)  sheet  is  a  time  sheet  used  to  measure  the  amount  of  time  that  you 
typically  spend  on  training  or  related  activities  during  your  duty  day.  Following  are  instructions  for 
‘ling  out  the  UN  I T  COMMANDER  WITS  sheet: 


I.  Enter  information  at  the  top  and  bottom  of  your  WITS  sheet. 

1.  Print  your  naroe--last,  first,  and  middle  initial. 

2.  Fill  in  your  workcenter  rsurber,  e.g.,  67th  CSG/PDHQ. 

3.  Fill  in  your  SSAN. 

4.  Fill  in  the  Week  Ending  date--the  last  day  of  the  applicable  work  week  or  time  period 
(might  bt-  UTA  weekend). 

5.  DO  NOT  fill  in  Data  Collector  blank. 

II.  Record  all  time  to  the  nearest  quarter  hour,  e.g.,  8.25  hours,  2.75  hours. 

III.  For  each  day  of  the  week,  fill  out  the  blocks  for  Numbers  1  through  9  as  they  apply.  Refer  to  the  in¬ 
structions  below,  as  necessary. 

Block  1:  Record  nuaber  of  hours  in  your  duty  day,  whether  it  is  the  nounal  8  hours  or  seme  different  amount 
of  time. 

Block  2:  Actual  time  spent  introducing  and  orienting  the  OJT  program  to  supervisors,  trainers,  and 

trainees. 

Block  3:  This  is  the  time  actually  spent  reviewing  and  judging  the  status  of  training  programs. 

Block  4:  Actual  time  spent  evaluating  progress  of  airmen  in  upgrade  training  who  have  reached  evaluation 
points  requiring  the  Cctrrrander's  involvement. 

Blockjj:  Actual  time  spept  evaluating  trairees  who  fail  to  progress  (Training  Status  Code  "T")  or  who  are 
not  recommended  for  further  upgrade  training  (Training  Status  Code  "0")  for  possible  separation  from  the 
service. 

SlSSKJ;5  Actual  time  spent  selecting  unit  personnel  to  perform  third-party  task  evaluations. 

Block_7:  Actual  time  spent  interviewing,  evaluating,  and  judging  the  potential  of  those  who  failed  man¬ 
datory  career  development  courses. 

Block_8:  Actual  time  spent  reviewing  the  documentation  and  judging  the  quality  of  third-party  task  evalua¬ 
tions. 

Block_9:  Actual  time  spent  reviewing  documented  evaluations  of  unit  personnel  from  outside  agencies. 


In  order  to  supply  accurate  information,  it  is  important  that  you  fill  out  the  WITS  sheet  DAILY  for-the 
applicable  time  period. 
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Weekly  Inventory  of  Time  Spent  (WITS) 

Unit  Training  Managers 


Wook  Number _  Data  Collector 


UNIT  TRAINING  MANAGER  WITS  SHEET  INSTRUCTIONS 


The  WITS  (Weekly  Inventory  of  Time  Spent)  sheet  is  a  time  sheet  used  to  measure  the  amount  of  time  that  you 
typically  spend  on  training  or  related  activities  during  your  duty  day.  It  is  understood  that  in  many  cases 
you  may  act  in  more  than  one  of  the  capacities  listed  on  the  WITS  sheet  (for  example  unit  training  manager 
and  trainer).  If  this  is  the  case,  it  will  be  necessary  to  fill  in  the  time  spent  in  each  of  the  roles  that 
you  play  in  OJT.  If  you  are  acting  in  more  than  one  role  at  a  time,  record  on  your  UITS  sheet  the  time 
spent  doing  tasks  principally  related  to  a  particular  OJT  role. 


Following  are  instructions  for  filling  out  the  UNIT  TRAINING  MANAGER  WITS  sheet: 

I.  Enter  information  at  the  top  and  bottom  of  your  WITS  sheet. 

1.  Print  your  name--last,  first,  and  middle  initial. 

2.  Fill  in  your  workcenter  number,  e.g.,  67th  CSG/PDKQ. 

3.  Fill  in  your  SSAN. 

4.  Fill  in  the  Week  Ending  date--the  last  day  of  the  applicable  work  week  or  time  period 
(might  be  UTA  weekend). 

5.  DO  NOT  fill  in  Data  Collector  btank  inless  you  have  been  assigned  that  duty. 

II.  Record  all  time  to  the  nearest  quarter  hour,  e.g.  8.25  hours,  2.75  hours. 

HI.  for  each  day  of  the  week,  fill  out  the  blocks  for  Numbers  1  through  8  as  they  apply.  Refer  to  the  in¬ 
structions  below,  as  necessary. 

8lock  1.  Record  nunber  of  hours  in  your  duty  day,  whether  it  is  the  normal  8  hours  or  some  different  amount 
of  time. 

Block  2.  These  activities  could  include  providing  orientations  to  supervisors  and  trainees,  overseeing 
completions  of  volume  review  exercises,  issuing  course  materials,  counseling  trainees  on  their  progress,  a-d 
monitoring  the  overall  progress  of  trainees  in  the  CDC  program. 

Block  3.  These  activities  may  include  unscheduled  evaluation  visits  to  workcenters,  formal  evaluation  of 
training  effectiveness,  and  preparation  of  reports  or  memoranda  for  record. 

Block  4:  Activities  may  include  helping  supervisors  determine  task  performance  requirements,  overall 
workcenter  requirements,  or  development  of  controls.  These  may  also  include  teaching  principles  of  training 
and  advising  on  means  to  solve  training  difficulties. 

Block  S:  Actual  time  spent  working  directly  with  the  management  of  the  upgrade  training  program  as  opposed 
to  qualification  training  or  other  programs  such  as  ancillary  training.  These  activities  may  include 
documenting  and  fulfilling  the  administrative  requirements  of  AFR50-23. 

Block  6:  Any  time  spent  with  these  agencies  resolving  training  issues,  scheduling  training,  arranging  test¬ 
ing  and  evaluation,  etc. 

Block  7:  Actual  time  spent  preparing  agendas,  conducting  meetings,  or  composing/documenting  minutes. 

Block  8:  Time  spent  performing  additional  duties  or  details,  either  formally  or  informally  assigned. 


In  order  to  supply  accurate  information,  it  is  important  that  you  fill  out  the  WITS  sheet  DAILY  for  the 
applicable  time  period. 
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SUPERVISOR.  EVALUATOR.  TRAINER  U1TS  SHEET  INSTRUCTIONS 


The  WITS  (Weekly  Inventory  of  Time  Spent)  sheet  is  a  time  sheet  used  to  measure  the  anoint  of  time  that  you 
typically  spend  on  training  or  related  activities  during  your  duty  day.  It  is  understood  that  in  many  cases 
you  may  act  in  more  than  one  of  the  capacities  listed  on  the  WITS  sheet  (for  example,  supervisor  and 
trainer).  If  chis  is  the  case,  it  will  be  necessary  to  fill  in  the  time  spent  in  each  of  the  roles  that  you 
play  in  OJT.  If  you  are  acting  in  mare  than  one  role  at  a  time,  record  on  your  WITS  sheet  the  time  spent 
doing  tasks  principally  related  to  a  particular  OJT  role. 


Following  are  instructions  for  filling  out  the  SUPERVISOR,  EVALUATOR,  TRAIHER  WITS  sheet: 

I.  Enter  information  at  the  top  and  bottom  of  your  WITS  sheet. 

1.  Print  your  name--last,  first,  and  middle  initial. 

2.  Fill  in  your  workcenter  number,  e.g.,  67th  CSG/PDMQ. 

3.  Fill  in  your  SSAN. 

4.  Fill  in  the  Week  Ending  date--the  last  day  of  the  applicable  work  week  or  time  period 
(might  be  UTA  weekend). 

5.  DO  NOT  fill  in  Data  Collector  blank  inless  you  have  been  assigned  that  duty. 

II.  Record  all  time  to  the  nearest  quarter  hour,  e.g.,  8.25  hours,  2.75  hours. 

III.  For  each  day  of  the  week,  fill  out  the  blocks  for  Numbers  1  through  15  as  they  apply.  Refer  to  the 
instructions  below,  as  necessary. 

Block  1:  Record  nunber  of  hours  in  your  duty  day,  whether  it  is  the  normal  8  hours  or  some  different  amount 
of  time. 


Bl£>ck__2:  These  activities  might  include  planning,  coordinating,  and  setting  the  date,  time,  and  place  for 

doing  OJT  on  a  task. 

Block  3:  These  activities  might  include  planning,  coordinating,  and  setting  the  date,  time,  and  place  for 

studying  CDCs  or  other  publications  supporting  OJT. 

Block  4:  The  actual  time  spent  reviewing  and  recording  task  training. 

Block  5:  The  actual  time  spent  reviewing  and  recording  knowledge  training. 

Block  6:  These  activities  include  the  actual  conduct  of  task  training  as  a  trainer. 

Block  7.  These  activities  include  the  actual  conduct  of  knowledge  training  as  a  trainer,  classroom  teacher. 

Block  8:  Actual  time  spent  evaluating  trainee  performance. 

Block_9:  Actual  time  spent  testing  or  reviewing  knowledge  training,  e.g.,  volume  review  exercises,  quizzes. 


Block  10:  Actual  time  spent  as  a  trainee  receiving  career  knowledge  training  through  classroom 
or  COC  activity. 


self-study. 


Block  11 :  Actual  time  spent  as  a  trainee  receiving  task-oriented  instruction,  practice,  and  evaluation, 
Block  12:  Actual  time  spent  as  a  supervisor  selecting  personnel  to  conduct  training. 


BLy.k  13:  Actual  time  spent  as  a  supervisor  setting  up  the  OJT  for  an  individual  or  group, 
tivities  may  include  determining  task  requirements,  reviewing  comoetence  of  a  newly- assigned 
revising  the  direction  of  the  individual  or  group's  OJT  program. 


These  ac- 
trainee,  or 


Block  14:  Actual  time  scent  receivina  auirtarv*A  fp«n  nr  nrAw«Wmn 


Block  15: 
adequate. 


Actual  time  spent  counseling  trainees  when  difficulties  arise  or  when  trainee  progress  is  in- 


In  order  to  supply  accurate  information,  it  is  important  that  you  fill  out  the  WITS  sheet  DAILY  for  the 
applicable  time  period. 


Figure  H.5-4 
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TRAINEE  UITS  SHEET  INSTRUCTIONS 


The  UITS  (Weekly  Inventory  of  Time  Spent)  sheet  is  a  time  sheet  used  to  measure  the  amount  of  time  that  you 
typically  spend  on  training  or  related  activities  during  your  duty  day.  It  is  understood  that  in  many  cases 
you  may  act  in  more  than  one  of  the  capacities  listed  on  the  WITS  sheet  (for  example,  supervisor  and 
trainee).  If  this  is  the  case,  it  will  be  necessary  to  fill  in  the  time  spent  in  each  of  the  roles  that  you 
play  in  OJT.  If  you  are  acting  in  more  than  one  role  at  a  time,  record  on  your  UITS  sheet  the  time  spent 
doing  tasks  principally  related  to  a  particular  OJT  role. 


Following  are  instructions  for  filling  out  the  TRAINEE  WI.TS  sheet: 

I.  Enter  information  at  the  top  and  bottom  of  your  UITS  sheet. 

1.  Print  your  name--last,  first,  and  middle  initial. 

2.  Fill  in  your  uorkcenter  number,  e.g.,  67th  CSG/PDMQ. 

3.  Fill  in  your  SSAN. 

4. '  Fill  in  the  Week  Ending  date--the  last  day  of  the  applicable  work  week  or  time  period 

(might  be  UTA  weekend). 

5.  DO  NOT  fill  in  Data  Collector  blank  unless  you  have  been  assigned  that  duty. 

II.  Record  all  time  to  the  nearest  quarter  hour,  e.g.,  8.25  hours,  2.75  hours. 

III.  For  each  day  of  the  week,  fill  out  blocks  for  Nunbers  I,  10,  11,  and  any  others  that  might  apply. 
Refer  to  the  instructions  below,  as  necessary. 

Block  1:  Record  nunber  of  hours  in  your  duty  day,  whether  it  is  the  normal  8  hours  or  some  different  amount 
of  time. 

Block  10:  Actual  time  spent  as  a  trainee  receiving  career  knowledge  training  through  classroom,  self-study, 
or  COC  activity. 

Block  11.:  Actual  time  spent  as  a  trainee  receiving  task-oriented  instruction,  practice  and  evaluation. 


In  order  to  supply  accurate  information,  it  is  important  that  you  fill  out  the  WITS  sheet  DAILY  for  the 
applicable  time  period. 
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GENERAL  PURPOSE,  > *  i* 


H . 6  BASE  CRIME  STATISTICS 


H.6.1  Data  To  Be  Collected.  The  67  SPS  will  send  AFHRL  the 

Base  Crime  Analysis  Report 
(Figure  H.6-1,  p.  41) 

each  quarter.  This  analysis  will  include  the  number  of  incidents 
in  the  following  categories:  Crimes  Against  Persons,  Crimes 
Against  Property,  Theft  of  Government  Property,  Theft  Of  Private 
Property,  and  Drug  Incidents. 

H.6.2  Requirements  For  DCR.  The  DCR  needs  no  special  background 
or  skills. 

H.6.3  Frequency  Of  Data  Collection.  Quarterly. 

H.6.4  Data  Collection  Procedure. 

-  POC  is  the  superintendent  of  the  67  SPS  Reports  and 
Analysis  Branch. 

-  1ST  support  will  come  from  TSgt  Rick  Booth,  X3070. 

-  After  receiving  the  crime  analysis  reports,  the  DCR  will 
obtain  a 

Security  Police  Crime  Analysis  Data  Collection  Form 
(Figure  H.6-2,  p.  42). 

If  the  crime  analysis  reports  are  not  received  by  the  7th 
of  the  month  following  their  issuance,  or  the  data  ap¬ 
pears  incomplete  or  inaccurate,  the  DCR  shall  coordinate 
a  personal  visit  with  the  Security  Police  Squadron  to  ob¬ 
tain  the  missing  information. 

-  The  DCR  will  look  at  each  category  of  crime  separately 
and  count  the  number  of  times  Security  Police  response 
was  made  to  that  specific  category. 

DCR  will  put  his  name  and  the  date  in  the  appropriate 
blank. 


H.6.5  Time  Requirements  For  Data  Collection.  Time  requirements 
for  the  collection  of  Base  Crime  Statistics  data  areas  follows: 
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0.25  Obtain  Data  Recording  Forms  and  Crime 

Analysis  Reports 

0.50  Record  data  onto  Data  Recording  Form 

0.50  Follow' up  actions  to  remedy  deficient  data 

0.07  Data  input 

H.6.6  Disposition  Of  Data.  After  checking  addition  for  ac¬ 
curacy,  the  DCR  will  deliver  the  data  recording  form  to  the  DIR, 
who  will  have  the  data  entered  into  the  Base  Crime  Statistics 
data  table. 
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DEPARTMENT  OF  THE  AIR  FORCE 

MEAOOUARTERS  47  TH  COK9AT  SUPPORT  OROUP (TAC) 
•ERGiTAGM  AIR  FORCE  SASC  TX  71743-5000 


RCPtr  TO 

ah* of  SPAR  (TSgt  Calderon,  4038)  01  AUG  86 

suiter  Crime  Analysis  for  the  Second  Quarter  of  1986  (APR-JUN) 


,0  Distribution  “A" 


1.  The  following  is  an  analysis  of  crime  and  incident  activities  on  Bergstrom 
AF8  for  the  second  quarter  of  1986.  Data  contained  in  this  report  indicates 
crimes  against  persons  or  property,  drug  offenses,  lost  or  stolen  property  and 
other  security  police  reponses. 

2.  CRIME  AGAINST  PERSONS:  Security  Police  responded  to  2  aggravated 
assaults,  3  simple  assaults,  2  domestic  disturbances,  1  obscene  phone  call,  2 
false  official  statements  and  2  child  neglect  cases. 


3.  CRIMES  AGAINST  PROPERTY:  Security  Police  responded  to  2  housebreakings,  1 
auto  theft  and  17  cases  of  vandalism.  Vandalism  increased  from  13  cases  last 
quarter  to  17  this  quarter.  Six  involved  government  property  and  11  involved 
private  property.  The  following  buildings  and  their  parking  lots  were  the 
targets  of  vandalism  incidents:  Dorms  2002,  2311,  2103,  2104,  2403,  BLDG 
1643,  2226,  3514,  4865,  parking  lots  of  2104,  2401,  2410,  2700,  and  239 
Anderson  Lane,  759-B  Robin  Lane,  5125  McWhirk  in  the  housing  area. 


4.  THEFT  OF  GOVERNMENT  PROPERTY:  There  were  12  thefts  this  quarter.  For 
t.ime  and  dates  of  each  occurrence  refer  to  attachment  1.  The  total  amount  of 
goverment  property  reported  stolen  is  51,549.72  with  31,297.32  recovered. 

5.  THEFT  OF  PRIVATE  PROPERTY:  There  were  46  reported  thefts  this  quarter,  of 
these,  11  were  shoplifting  cases.  The  total  amount  of  private  property  stolen 
is  ^19,641.08  with  510,130.69  recovered. 


6*  There  were  15  drug  related  incidents  this  quarter.  Six  were  at 

the  main  gate,  7  were  at  the  commercial  gate,  1  at  the  visitors  reception 
center  parking  lot,  and  1  at  the  commissary  parking  lot. 


ROBERT  M.  COLLINS,  Major,  USAF 
Chief  of  Security  Police 


5  Attachment 

1.  Larceny  of  Gov't  Property 

2.  Larceny  of  Private  Property 

3.  Shoplifting  Cases 

4.  Gov't/Private  Property  Thefts  by  Week 

5.  Bar  Graph  on  Drug  Abuse  Incidents 


ti  ouz 


.  J*»  * 


Figure  H.6-1 
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Figure  H.6-2 
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H. 7  MDC/MILAP  480  REPORT  COLLECTION  PROCEDURES 

H.7.1  Data  To  Be  Collected.  The  MDC  and  MILAP  480  Reports  are 
generated  by  the  67th  TRW  DCM/MASA  office.  These  reports  provide 
a  detailed  tracking  of  maintenance  activities  performed  by  the 
various  workcenters  at  Bergstrom  AFB.  These  reports  are  only 
generated  for  the  Active  component. 

H.7.2  Requirements  For  DCR.  Both  a  DCR  and  a  subject  matter  ex¬ 
pert  (SME)  are  required.  The  data  collector  will  not  require  any 
specific  experience  background.  The  SME  will  require  background 
in  either  426X2  or  431X1  AFSs. 

H.7.3  Frequency  Of  Data  Collection.  Data  from  the  MDC  and  MILAP 
480  reports  will  be  collected  monthly. 

H.7.4  Data  Collection  Procedure 


H.7.4.1  Data  Locations.  The  MDC  and  MILAP  480  reports  are  sent 
by  the  67th  TRW  DCM/MASA  office  to  AFHRL/OL-AK  in  Trailer  T-l. 


H.7.4. 2  MDC  Potential  Repeat  Action  Identification.  The  MDC 
and  MILAP  480  reports  will  be  provided  by  DCM/MASA  upon  the 
request  of  the  DCR.  The  POC  and  document  location  are: 


Building: 

Room: 

POC: 

Ext: 


1603 

18 

MSgt  Winkleman 
3348/9 


The  DCR  will  request  and  obtain  five  MDC  and  one  MILAP  480 
Report  from  the  location  listed  above.  It  takes  approximately 
two  to  three  days  for  NASA  to  generate  the  reports  once  they  have 
been  requested.  The  reports  will  cover  the  following  areas: 

1)  431X1  within  the  12th  AMU  for  A  flight 

2)  431X1  within  the  12th  AMU  for  B  flight 

3)  426X2  within  the  12th  AMU  for  Specialist  Flight 

Propulsion 

4)  426X2  within  67th  CRS  Jet  Engine  Intermediate 
Maintenance 

5)  431X1  within  the  67th  EMS  Phase  shop  Inspection 

Once  he  has  the  reports,  DCR  will  review  each  report  to 
identify  potential  repeat  maintenance  actions.  The  potential 
repeat  maintenance  actions  may  be  determined  in  the  following 
manner 

(See  Figure  H.7-1,  p.  48): 
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Determine  the  workcenter  to  which  each  report  applies. 


-  Locate  the  PWC  column.  Use  the  following  table  to  iden¬ 
tify  which  report  is  associated  with  which  workcenter. 

PWC  WORKCENTER 

12th  AMU  A  FLIGHT  (431X1) 

12th  AMU  B  FLIGHT  (431X1) 

12th  AMU  SPECIALIST  FLIGHT  (426X2) 
67th  EMS  MAINTENANCE/ PHASE  (431X1) 
67th  CRS  JEIM  (426X2) 

-  Write  the  workcenter  name  on  the  front  of  the  MDC  com¬ 
puter  report.  It  is  necessary  to  identify  the  workcenter 
with  the  reports.  The  purpose  is  to  track  how  many  main¬ 
tenance  repeat  actions  occur  for  the  five  shops: 

1)  12th  AMU  (431X1) ,  A  Flight 

2)  12th  AMU  (431X1),  B  Flight 

3)  12th  AMU  (431X1),  Specialist  Flight 

4)  67th  CRS  JEIM  (426X2) 

5)  67th  EMS  MAINTENANCE/PHASE  (431X1) 

-  To  accomplish  this,  the  DCR  will  review  the  contents  of 
the  columns  labeled: 

1)  WUC  (work  unit  code) 

2)  JCN  (job  control  number) 

3)  ID  (engine  serial  or  aircraft  tail  number) 

4)  HM 

5)  UNITS 

-  The  DCR  will  review  these  columns  and  highlight  those 
records  which  meet  the  following  criteria: 


a) 

ID 

= 

Same 

b) 

WUC 

= 

Same,  unless  first  digit  =  0 

c) 

JCN 

= 

Different' 

d) 

HM 

= 

Same 

e) 

UNITS 

= 

Consecutive  Is 

f) 

DAY 

= 

TBD  by  SME 

If  these  conditions  are  met 

(See  Figure  H.7-2,  p.  49), 

then  highlight  that  record  (row) .  Proceed  through  each 
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DG111 

DG112 

DG501 

DE210 

DR210 


1 


of  the  documents  looking  for  records  to  meet  these  test 
conditions.  Another  example  of  possible  repeat  mainten¬ 
ance  actions  is  shown  in 

Figure  H.7-3,  p.  50. 

An  example  of  a  situation  that  would  not  qualify  is  shown  in 
Figure  H.7-4,  p.  51. 

The  situation  would  not  qualify  because  even  though  both  actions 
have  the  same  WUC  and  different  JCNs,  they  were  accomplished  on 
different  engines  (IDs) . 

Once  the  repeat  maintenance  actions  have  been  highlighted, 
the  data  collector  will  give  the  reports  to  the  SME. 

H.7.4.3  Identification  Of  Actual  Repeat  Maintenance  Actions  On 
The  MDC.  The  SME  will  review  the  highlighted  actions  and  indi¬ 
cate  those  items  he/she  believes  are  repeat  actions  by  placing  an 
asterisk  in  the  right-hand  margin.  When  the  SME  is  finished,  the 
DCR  will  obtain  the  reports  and  count  the  number  of  asterisks  in 
each  report.  The  total  number  of  repeat  actions  for  the  five 
specific  shops: 

1)  12th  AMU  A  and  B  Flights  (431X1) :  two  MDC  reports 
added  together 

2)  12th  AMU  (431X1),  Specialist  Flight  (426X2) 

3)  67th  CRS  JEIM  (426X2) 

4)  67th  EMS  Maintenance  shop  (431X1)  PWC=DE210 
added  together 

These  totals  will  be  recorded  using  the 

MDC/MILAP  Data  Collection  Form — Figure  H.7-5,  p.  52. 

H.7.4.4  MDC  Data  Disposition.  The  DCR  will  provide  the  summary 
data  collection  memo  to  the  DIR. 

The  DIR  will  record  the  number  of  repeat  maintenance  actions  for 
the  month  for  each  workcenter  in  the  appropriate  data  table. 

H.7.4.5  MILAP  480  Potential  Repeat  Action  Identification.  The 
MILAP  480  reports  will  be  provided  by  DCM/MASA  upon  request  of 
the  DCR.  tHE  POC  and  document  location  is: 

Building:  1603 
Room:  18 
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POC:  MSgt  Winkleman 

Ext:  X3348/9 

The  DCR  will  obtain  \  monthly  MILAP  480  Report  from  the  location 
listed  above.  It  takes  approximately  two  to  three  days  for  MASA 
to  generate  the  report  once  it  has  been  requested. 

The  MILAP  480  Report,  Figure  H.7-6,  p.  53, 

is  used  as  an  additional  measure  for  obtaining  an  accounting  of 
repeat  maintenance  actions.  In  this  respect,  it  is  similar  to 
the  MDC  reports. 

The  major  differences  between  the  MILAP  480  and  the  MDC  reports 
are: 

1)  MILAP  480  pertains  only  to  AMU  (431X1  or  specialist 
flight)  actions 

2)  The  formats  are  different 

3)  MILAP  480  identifies  actions  at  the  aircraft  level  per 
pilot  reports. 

After  the  report  is  obtained,  the  DCR  will  highlight  those  items 
which  may  indicate  repeat  maintenance  actions. 

In  order  to  highlight  repeat  maintenance  actions,  the  DCR  must 
look  at  two  columns  of  information  on  the  report.  The  two 
columns  are: 

1)  Sortie  Seq-NBR 

2)  Rep-Rec 

The  Sortie-Seq-NBR  column  identified  which  maintenance  unit  serv¬ 
ices  the  aircraft.  This  study  is  concerned  with  those  Sortie- 
Seq-NBRs  that  begin  with  a  two  (2).  The  two  (2)  identifies  the 
12th  AMU. 

The  Rep-Rec  identifies  the  type  of  occurrence.  This  study  is 
only  concerned  with  those  incidents  coded: 

1)  C,  which  is  recurring  discrepancy 

2)  R,  which  is  repeat  discrepancy 

3)  A,  which  is  repeat  plus  abort 

4)  B,  which  is  recurring  plus  abort 

Thus,  the  two  criteria  the  DCR  uses  to  highlight  a  record  are: 
a)  that  the  Sortie-Seq-NBR  begins  with  a  2  or  a  4,  and 
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b)  that  the  Rep-Rec  code  be  a  C,  R,  A,  or  B. 

Once  the  DCR  has  highlighted  the  MILAP  480  Report,  the  SME  will 
examine  the  report  and  identify  actual  repeat  maintenance  ac¬ 
tions. 


H.7.4.6  Identification  Of  Actual  Repeat  Maintenance  Actions  On 
The  MILAP  480  Report.  The  SME  will  provide  expertise  in  making  a 
final  assessment  determining  whether  highlighted  actions  are  in 
fact  repeat  maintenance  actions.  If  the  SME  decides  that  the  ac¬ 
tions  are  repeat  maintenance  actions,  he  will  place  an  asterisk 
(*)  in  the  right-hand  column. 

The  DCR  will  then  count  up  the  number  of  asterisks  for  the 
12th  AMU  and  write  that  total  in  the  Repeat  Maintenance  Count 
block  on  the  Data  Collection  Form. 

Examples  of  correct  and  incorrect  highlighted  repeat  actions 
are  shown  in 

Figure  H.7-7,  p.  54. 

The  first  highlighted  event  is  correct,  i.e.  Sort-Seq-NBR  is  217 
and  the  Rep-Rec  code  is  B.  The  second  event  highlighted  is  in¬ 
correct  because  the  Sort-Seq-NBR  is  721,  even  though  the  Rep-Rec 
is  an  R.  Figure  H.7-6  (p.  53)  presents  another  example  of  a 

correctly  highlighted  page. 

H.7.5  Time  Requirements .  The  DCR  will  ask  for,  receive,  and 
review  this  data  on  a  monthly  basis.  The  time  requirement  for 
the  collection,  evaluation  and  recording  of  data  is  as  follows: 

MDC/MILAP  480  Report  Monthly  Collection 

0.25  Arrange  report  generation/pick-up 

0.25  Miscellaneous  travel  on-base 

0.16  Pick-up  reports 

1*50  Data  collector  highlights  report 

0.75  1ST  SME  reviews  highlighted  items 

0.50  Extract  data  from  reporc 

0.12  Input  data 

MDC/MILAP  480  Data  Disposition.  When  the  MILAP  480 
Report  has  been  completed  and  the  asterisk  count  included  in  the 
MDC/MILAP  480  Report  Data  Recording  Form,  the  form  will  be 
provided  to  the  DIR. 

The  DIR  will  enter  the  MILAP  480  repeat  incident  count  into  the 
data  table. 
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Figure  H.7-1 
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MDC/MilAP  Data  Collection  Form 
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Figure  H.7-5 
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Figure  H.7-6 
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Figure  H.7-7 


H.  8 


BASELINE  SURVEYS 


H.8.1  Data  To  Be  Collected .  The  Prototype  AOTS  Baseline  Surveys 
are  survey  instruments  that  are  distributed  to  all  workcenter 
personnel  prior  to  the  start  of  SLT&E.  The  surveys  are  designed 
to  measure  the  current  OJT  environment  concerning  the  level  of 
computer  experience,  views  and  opinions  regarding  training  and 
OJT,  and  the  availability  of  operational  equipment  for  use  in  the 
OJT  process.  There  are  three  types  of  Baseline  Surveys,  each 
tailored  to  a  particular  audience: 

1)  Commanders — Figure  H.8-1,  p.  57 

2)  Training  Managers — Figure  H.8-2,  p.  61 

3)  Supervisors,  Evaluators,  Trainers,  and  Trainees 

Figure  H.8-3,  p.  69 


H.8.2  Requirements  For  DCR.  The  DCR  who  administers  the 
Baseline  Surveys  does  not  require  a  specific  experience  back¬ 
ground  . 

H.8.3  Frequency  Of  Data  Collection.  Baseline  Surveys  will  be 
administered  once  near  the  end  of  the  baseline  data  acquisition 
period,  at  all  AOTS  and  control  workcenters. 

H.8.4  Data  Collection  Procedure.  For  POCs,  participants,  and 
locations  for  the  administration  of  Baseline  Surveys,  refer  to 
Attachments  3  and  4  at  the  end  of  Appendix  H  and  (for  control 
workcenters)  Attachments  at  the  end  of  Appendix  J. 

The  following  steps  must  be  followed  in  order  to  administer  the 
Baseline  Survey: 

-  Brief  all  OJT  and  unit  training  managers  on  the 
purposes  and  uses  of  the  Baseline  Surveys  and  the 
administration  and  collection  procedures. 

-  Identify  workcenter  commanders;  training  managers; 
and  supervisors,  evaluators,  trainers,  and  trainees. 

-  Photocopy  sufficient  copies  of  the  survey  for  the 
intended  workcenter. 

-  Assemble  surveys  and  answer  sheets  for  each  person. 
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-  Attach  appropriate  name  labels  to  each  survey. 

-  Group  them  according  to  component  and  workcenter. 

-  Give  survey  packets  to  the  appropriate  OJT  manager 
who  will  give  them  to  each  unit  training  manager  for 
distribution,  completion,  and  return  to  the  OJT 
manager . 

H.8.5  Time  Required  For  Data  Collection.  The  time  requirement 
for  the  administration  and  collection  of  Baseline  Surveys  is  as 
follows: 


Baseline  Survey  administration  and  collection  (One  workcenter  at 
Bergstrom) 


1  00  Brief  training  managers 

1-25  Copy  and  prepare  surveys  for  distribution 

0.25  Miscellaneous  travel  on-base 

Baseline  Survey  administration  and  collection  (One  workcenter  at 
Ellington) 


l.oo  Brief  training  managers 

0.50  Review  survey  questions 

0.25  Copy  and  prepare  surveys  for  distribution 

0-25  Miscellaneous  travel  on-base 


Disposition  Of  Data.  After  the  Baseline  Surveys  have  been 
administered,  the  DCR  will  deliver  the  completed  surveys  to  the 
CTM,  who  will  have  the  results  scanned  by  AFHRL/TS  and  entered 
into  the  Baseline  Survey  data  tables. 


PROTOTYPE  AOTS  BASELINE  SURVEY 
UNIT  COMMANDERS 


PRIVACY  ACT  STATEMENT 


AUTHORITY: 

PRINCIPAL 


44  USC  3101:  10  USC 
PURPOSE:  ^he  SSAN 


801?  AND  E0  0307 
will  be  used  to 


_ _  identify  personnel  participating 

ROuifi^ul^S:™  P1?tToy§etermine  the  effectiveness  of  the  current  OJT  program 
2.  to  determine  the  effectiveness  of  th 


in  the  Advanced  On-the-job 


DISCLOSURE  IS  HAND, 
to  other  personae 
requiced  monitoring 
form  its  mission. 


.  To  determine 
'ATOPY:  the. 

I  included.  i._ 
of  training  relat 


mandatory  to  malce  pgsPtive  identification  of.  individuals  .in  relation 
— .  Failure  to  provide  .  this,  information  will  prevent 

will  jeopardize  tne  ability  of  the  AOTS  program  to  per- 


SSAN  is 

in  KrasrasaPi 


DIRECTIONS:  On  the  accompanying  answer  sheet,  please  enter  your  name  (in  the  "Name 
Grid"  block)  and  SSAN  (in  the  "Numeric  Grid"  block,  columns  1  through  9).  Indicate  your 
response  to  each  question  by  filling  in  the  corresponding  oval  for  each  question. 


Note:  Computer  use  does  not  include  the  use  of  a  hand  calculator. 


1.  Have  you  ever  used  a  computer? 
a  -  Yes  b  -  No 

IF  YOU  ANSWERED  NO.  GO  TO  QUESTION  NUMBER  7. 


2 .  Do  you  use  a  computer  in  ycur  j  ob? 

a  -  Yes  b  -  No 

3.  Have  you  ever  owned  a  computer? 

a  -  Yes  b  -  No 

4.  How  comfortable  do  you  feel  using  a  computer? 

a  -  Very  comfortable 
b  -  Comfortable 

c  -  Neither  comfortable  nor  uncomfortable 
d  -  Uncomfortable 
e  -  Very  uncomfortable 

5.  Have  you  ever  received  instruction  where  the  information  was 
presented  on  a  computer? 

a  -  Yes  b  -  No 

6.  How  would  you  rate  your  computer-based  learning  experience? 

a  -  N/A,  haven't  received  any  instruction  through  a  computer 
b  -  Outstanding 
c  -  Excellent 
d  -  Satisfactory 
e  -  Marginal 
f  -  Unsatisfactory 
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7.  Have  you  ever  taken  any  computer  classes? 

a  -  Yes  b  -  No 

8.  How  often  do  you  play  video  games? 

a  -  Daily 
b  -  Weekly 
c  -  Monthly 
d  -  Yearly 
e  -  Never 

9.  What  is  your  rank? 

a.  2d  -  1st  Lt 

b.  Captain 

c.  Major 

d.  Lt  Colonel 

e.  Colonel 

10.  To  what  type  of  unit  are  you  assigned? 

a.  Maintenance 

b.  Security  Police 

c.  Personnel 

d.  Other 

11.  How  long  have  you  been  commander  for  your  present  unit? 

a.  Less  than  3  months  e.  12  but  less  than  18 

months 

b.  3  but  less  than  6  months  f.  18  but  less  than  24 

months 

c.  6  but  less  than  12  months  g.  24  but  less  than  36 

months 

d.  9  but  less  than  12  months  h.  36  months  or  more 

Using  the  following  scale,  indicate  how  important  you  feel  train¬ 
ing  is  (in  general,  not  to  you  particularly)  to  each  of  the  areas 
given  below. 


NOT  IMPORTANT  MODERATELY  EXTREMELY 

AT  ALL  IMPORTANT  IMPORTANT 

a - b - c - d - e - f - g 

12.  Retention  beyond  the  first  term 

13.  Initial  decision  to  enter  the  Air  Force 
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14.  Job  satisfaction 


15.  Safety 

16.  Morale/Motivation 

17.  Accomplishment  of  the  mission 


18.  How  do  you  feel 

following  scale.) 

about  automating  the  OJT 

system? 

(Use  the 

DON'T  FAVOR 

NEUTRAL 

FAVOR 

IT  AT  ALL 

IT  A  LOT 

a - b - 

- c - d - e — 

- f- 

-  g 

19.  How  satisfied  are  you  with  the  current  OJT  program? 

a.  very  satisfied 

b.  satisfied 

c.  somewhat  satisfied 

d.  neither  satisfied  nor  dissatisfied 

e.  somewhat  dissatisfied 

f.  dissatisfied 

g.  very  dissatisfied 

20.  Are  you  experiencing  any  problems  with  your  OJT  program? 

a.  no 

If  yes,  in  the  following  areas:  (Mark  each  that  applies.) 

b.  OJT  administration 

c.  quality  of  trainers 

d.  training  materials 

e.  evaluation  materials 

f.  quality  of  evaluators 

g.  training  managers 


Use  the  following  scale  for  questions  21-33. 


STRONGLY 

DISAGREE 


NEUTRAL 


STRONGLY 

AGREE 


21.  Air  Force  policies  governing  the  OJT  program  provide  me  with 
the  flexibility  I  need  as  a  commander  to  operate  my  OJT  program. 

22.  On-the-job  training  has  a  major  impact  on  unit  effectiveness. 

23.  My  unit  has  the  necessary  training  and  evaluation  materials 
needed  to  support  my  mission. 

24.  My  unit  has  the  necessary  operational  training  equipment  to 
perform  my  OJT  mission. 

25.  My  unit  training  requirements  are  accurately  defined. 

26.  I  can  quickly  extract  training  information  from  my  unit 
training  records  to  assess  unit  training  status. 

27.  My  unit  training  records  accurately  reflect  my  unit's 
qualifications . 

28.  My  unit  training  records  allow  me  to  accurately  assess 
trainees'  accomplishments. 

29.  I  have  the  necessary  training  management  indicators  to  ef¬ 
fectively  manage  my  OJT  program. 

30.  My  unit  has  the  necessary  training  development  tools  and  ex¬ 
pertise  to  develop  training  materials. 

31.  I  am  very  confident  the  current  OJT  program  qualifies  my 
airmen  in  the  proper  tasks  for  my  mission. 

32.  My  OJT  trainers  are  well  qualified. 

33.  I  am  satisfied  with  the  time  it  takes  to  qualify  a  trainer 
in  a  position. 

34.  What  problems  are  you  experiencing  with  your  OJT  program? 

35.  How  would  you  improve  your  OJT  program? 


Figure  H.8-1 
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PROTOTYPE  AOTS  BASELINE  SURVEY 
OJT  MANAGERS 


AUTHORITY:  44  USC  3101: ,10  USC 
PRINCIPAL  PURPOSE:  ^he  SSAN 


PRIVACY  ACT  STATEMENT 

be  EusecPto  identify  personnel  participating  in  the  Advanced  On-the-j'.b 


to  Qther  personnel  included,  in  t 
required  mpnijonng  of  training  re 
form  jts  mission. 


iH  is  mandatory  to  m 
the  AOIS  prototype, 
ilated  items  which  wi 


, ,  F.ai  lure,,  to  pi 
1 1  jeopardize  ttv 


e  ability 


i  of.  individuals  .in  relation 
is,  information  will  prevent 
of  the  AOTS  program  to  per- 


DIRECTIONS:  On  the  accompanying  answer  sheet,  please  enter  your  name  (in  the  "Name 
Grid"  block)  and  SSAN  (in  the  "Numeric  Grid"  block,  columns  1  through  9).  Indicate  your 
response  to  each  question  by  filling  in  the  corresponding  oval  for  each  question. 


Note:  Computer  use  docs  not  include  the  use  of  a  hand  calculator. 

1.  Have  you  ever  used  a  computer? 
a  -  Yes  b  -  No 


2.  Do  you  use  a  computer  in  your  job? 
a  -  Yes  b  -  No 


3 .  Have  you  ever  owned  a  computer? 
a  -  Yes  b  -  No 


4.  How  comfortable  do  you  feel  using  a  computer? 
a  -  Very  comfortable 
b  -  Comfortable 

c  -  Neither  comfortable  nor  uncomfortable 
d  -  Uncomfortable 
e  -  Very  uncomfortable 


5.  Have  you  ever  received  instruction  where  the  information  was 
presented  on  a  computer? 

a  -  Yes  b  -  No 


6.- 


WfMJ  Wnill  H  ^rnn  >*a4* 

**  —  •'  ••  *'—■*■'*  j  VU  4.U  V 

a  -  N/A,  haven't 
b  -  Outstanding 
c  -  Excellent 


:  your  computer— based  learning  experience? 
received  any  instruction  through  a  computer 


d  -  Satisfactory 
e  -  Marginal 
f  -  Unsatisfactory 
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7. 


Have  you  ever  taken  any  computer  classes? 
a  -  Yes 
b  -  No 

8 .  How  often  do  you  play  video  games? 
a  -  Daily 
b  -  Weekly 
c  -  Monthly 
d  -  Yearly 
e  -  Never 


PLEASE  USE  THE  FOLLOWING  DEFINITIONS  TO 
ANSWER  QUESTIONS  ON  THE  FOLLOWING  PAGES: 


MANAGING  UPGRADE  TRAINING  PROGRAM:  Work  directly  associated  with 
the  status  of  personnel  in  upgrade  training.  Examples  are  main¬ 
taining  the  Monthly  OJT  Roster,  entering  personnel  into  training, 
adjusting  training  dates,  processing  skill  level  upgrades  and 
withdraws  from  training,  and  performing  upgrade  trainee  orienta¬ 
tions. 

CONDUCTING/REPORTING  STAFF  ASSISTANCE  VISITS:  All  activities 
supporting  planning  for,  actually  conducting,  and  documenting 
both  informal  and  formal  staff  assistance  visits,  when  required. 

MANAGING  THE  CPC  PROGRAM;  Ensuring  proper  CDCs  are  ordered  for 
upgrade  trainees,  processing  reenrollments  and  enrollment  exten¬ 
sions,  and  ensuring  other  administrative  CDC  requirements  are 
met,  as  required. 

COORDINATING  WITH  CBPO  AND  OUTSIDE  AGENCY  FUNCTIONS:  Working 
with  CBPO  concerning  on-the-job  training  matters  effecting  promo¬ 
tions,  classification,  retraining,  discharges/separations,  etc. 
Coordinating  (scheduling  people  and  places)  with  FTD,  Base  Educa¬ 
tion,  etc. 

ASSISTING  SUPERVISORS/COMMANDERS :  Providing  assistance  to 
develop  and  use  training  plans,  identify  training  requirements, 
understand  and  use  the  ISD  process,  select  task  evaluators,  and 
provide  advice  on  other  OJT  matters. 

ORGANIZING/CONDUCTING  MEETINGS:  Preparation  for,  conducting,  and 
documenting  required  OJT  meetings. 

DAILY  SUPERVISORY  ACTIVITIES:  Normal  supervision  and  management 
of  workcenter  personnel. 
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ADMINISTRATIVE  ACTIVITIES :  Routine  processing  of  computer  system 
transactions  and  normal  paperwork  flow  through  the  workcenter 
such  as  read  files,  memos  for  record,  forms  preparation,  answer¬ 
ing  telephones  and  taking  messages,  etc. 


9.  What  is  your  rank? 


a. 

Amn-SrA 

e. 

MSgt 

b. 

Sgt 

f. 

SMSgt 

c. 

SSgt 

g- 

CMSgt 

d. 

TSgt 

10.  What  is  your  total  time  in  service? 


a. 

Less  than  2  years 

g- 

12 

but 

less 

than 

14 

years 

b. 

2 

but 

less  than  4 

years 

h. 

14 

but 

less 

than 

16 

years 

c. 

4 

but 

less  than  6 

years 

i. 

16 

but 

less 

than 

18 

years 

d. 

6 

but 

less  than  8 

years 

j- 

18 

but 

less 

than 

20 

years 

e. 

'  8 

but 

less  than  10  years 

k. 

20 

years  or 

more 

f. 

10  but 

:  less  than 

12  years 

11.  Row  long  have  you  been  an  OJT  Manager? 


a. 

Less  than  6  months 

e. 

6-7  years 

b. 

6  months-1  year 

f . 

8-9  years 

c. 

2-3  years 

g- 

10  years  or  more 

d. 

4-5  years 

12.  Which  of  the  following  best  describes  your  OJT 
Management  duties? 


a. 

Additional 

duty 

d. 

Full-time,  Maintenance 

b. 

Full-time, 

Base  OJT 

Training 

c. 

Full-time, 

Unit  OJT 

e. 

Other 

13 .  What  do  you  feel  is  the  most  important  function  of  an  OJT 
Manager? 


a. 

b. 
c  * 

d. 

e. 

f. 

g- 

h. 

i. 


Managing  upgrade  training  program 
Conducting/Reporting  staff  assistance  visits 


Marvarfi 

*  mi  4.1IVJ  V11W- 


pxwv^i.aui 


Coordinating  with  CBPO  and  outside  agency  functions 
Assisting  supervisors/commanders 
Organizing/Conducting  meetings 
Daily  supervisory  activities 


Administrative  activities 


Other 
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14.  What  do  you  feel  is  the  least  important  function  of  an  OJT 
manager? 

a.  Managing  upgrade  training  program 

b.  Conducting/Reporting  staff  assistance  visits 

c.  Managing  the  CDC  program 

d.  Coordinating  with  CBPO  and  outside  agency  functions 

e.  Assisting  supervisors/commanders 

f.  Organizing/Conducting  meetings 

h.  Daily  supervisory  activities 

i.  Administrative  activities 

j .  Other 

15.  If  you  had  more  time  to  spend  on  one  aspect  of  your  job, 
what  would  it  be? 

a.  Managing  upgrade  training  program 

b.  Conducting/Reporting  staff  assistance  visits 

c.  Managing  the  CDC  program 

d.  Coordinating  with  CBPO  and  outside  agency  functions 

e.  Assisting  supervisors/commanders 

f.  Organizing/Conducting  meetings 

g.  Daily  supervisory  activities 

h.  Administrative  activities 

i .  Other 

16.  What  do  you  see  as  the  area  within  the  scope  of  OJT  needing 
the  most  attention  or  resources? 

a.  Managing  upgrade  training  program 

b.  Conducting/Reporting  staff  assistance  visits 

c.  Managing  the  CDC  program 

d.  Coordinating  with  CBPO  and  outside  agency  functions 

e.  Assisting  supervisors/commanders 

f.  Organizing/Conducting  meetings 

g.  Daily  supervisory  activities 

h.  Administrative  activities 

i .  Other 

USE  THE  PERCENTAGE  SCALE  BELOW  TO  ANSWER  QUESTIONS  17-25. 


r* 

u  • 

zero 

h. 

35 

o. 

70 

b. 

5 

i. 

40 

P- 

75 

c. 

10 

j- 

45 

q. 

80 

d. 

15 

k. 

50 

r. 

90 

e. 

20 

1. 

55 

s. 

100 

f. 

25 

m. 

60 

g* 

30 

n. 

65 
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17.  On  the  average,  what  percent  of  your  on-the-job  time  do  you 
spend  in  training-related  activities? 

Consider  your  answer  to  question  17  as  a  block  of  time.  Nov/  in¬ 
dicate  how  this  time  is  distributed  among  each  activity  listed 
below.  Your  answers  will  be  the  relative  percent  of  your  train¬ 
ing  time  you  spend  on  each  activity.  The  total  of  questions  18-25 
should  equal  100%.  It  may  be  helpful  to  first  write  in  your 
numerical  percentages  directly  on  the  survey  in  the  spaces 
provided.  Then,  using  the  scale  above,  enter  the  appropriate 
letter  response  on  your  answer  sheet.  Be  sure  your  responses 
equal  100%  Refer  to  the  previous  page  for  definitions  of  each 
activity. 

13.  Managing  upgrade  training  program  _ 

19.  Conducting/Reporting  staff  assistance  visits  _ 

20.  Managing  the  CDC  program  _ 

21.  Coordinating  with  CBPO  and  outside  agency  functions  _ 

22.  Assisting  supervisors/commanders  _ 

23.  Organizing/Conducting  meetings  _ 

24.  Daily  supervisory  activities  _ 

25.  Administrative  activities 


THIS  IS  A  DESCRIPTION  OF  AN  AUTOMATED  TRAINING  SYSTEM.  Consider 
the  system  to  be  fully  operational.  No  knowledge  of  computer 
programming  or  computer  languages  are  necessary  to  use  the  sys¬ 
tem.  Delivery  is  over  existing  computer  terminals.  This  system 
is  used  with  other  computer  systems  in  the  Air  Force,  like  those 
within  personnel  or  maintenance. 


POSITION-LEVEL. TASK  INFORMATION:  The  focus  of  the  training  sys¬ 
tem  is  on  the  specific  tasks  performed  in  each  position  in  a 
workcenter  or  shop.  Along  with  a  description  of  each  task  are 
given  the  knowledges  and  skills  needed  to  perform  the  task,  the 
standard  for  successful  performance,  a  suggested  training  method, 
and  a  recommended  order  for  training  the  tasks.  In  addition, 
other  types  of  training  requirements  are  identified.  For  ex¬ 
ample,  ancillary  training,  CDCs,  or  training  for  additional 
duties. 
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COMPUTERIZED  TRAINING  RECORD:  A  training  record  on  each  airman 
is  kept  on  file  in  the  computer  and  contains  a  list  of  what  the 
airman  has  been  trained  on,  the  training  method  used,  the  date 
the  training  was  completed,  and  the  scores  for  any  tests  or  per¬ 
formance  checks.  Supervisors,  commanders  and  OJT  managers  can 
review  these  training  records  at  any  time.  They  use  the  up-to- 
date  information  from  both  the  airman's  training  record,  and  the 
position  task  requirements  to  identify  where  additional  training 
may  be  needed. 

AUTOMATIC  SCHEDULING  AND  UPDATING:  Once  an  airman's  training 
requirements  are  identified,  a  training  plan  is  automatically 
generated  and  he/she  is  scheduled  for  the  training.  Portions  of 
the  training  are  available  to  be  conducted  "on  line",  that  is  the 
trainee  follows  lessons  and  takes  tests  directly  on  the  computer. 
Computer  programs  exist  within  the  system  to  assist  supervisors 
in  developing  more  localized  training  lessons  and  evaluation 
materials  should  they  be  necessary.  When  a  lesson  is  completed, 
the  date  and  test  results  are  automatically  recorded  in  the 
airman's  computerized  training  record.  Other  tasks  are  trained 
"off-line",  or  away  from  the  computer.  In  this  case,  supervisors 
record  their  evaluations  of  trainee  performance  on  special 
computer-read  forms  so  the  information  can  be  automatically  en¬ 
tered  into  the  airman's  record. 

THIRD-PARTY  EVALUATIONS :  Sometimes  an  airman's  performance  on 
tasks  is  evaluated  by  persons  other  than  his/her  trainer.  These 
"third  party"  evaluations  are  scheduled  automatically,  depending 
on  several  things,  like  the  importance  or  difficulty  of  the  task. 
The  dates  and  results  of  these  performance  checks  are  automati¬ 
cally  entered  into  the  individual  training  record  using  computer- 
read  forms  like  those  for  other  "off-line"  evaluations. 

STANDARDIZED  TRAINING  PRACTICES;  This  automated  training  system 
standardizes  many  aspects  of  the  training  process.  Whenever 
tasks  are  the  same  (on  one  base  or  between  bases)  the  methods 
used  to  train  individuals,  evaluate  their  proficiency,  and  docu¬ 
ment  the  results  should  be  the  same.  The  "third  party"  evalua¬ 
tions  also  help  insure  trainees  are  performing  tasks  ap¬ 
propriately. 

OTHER  USES  of  THE  TRAINING  INFORMATION:  The  training  information 
stored  m  the  computer  system  is  used  in  several  additional  ways. 
For  example,  managers  can  identify  the  tasks  for  a  specific  posi- 
tion  and  then  identify  all  the  persons  who  are  currently 
qualified  to  perform  the  tasks.  This  information  can  be  used  to 
help  select  the  best  person  for  a  job  or  TDY,  select  persons  for 
specialized  training  or  to  attend  PME  courses,  and  for  reporting 
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unit  readiness.  In  addition,  this  information  can  help  uncover 
areas  where  many  people  need  training  leading  to  a  new  training 
program  or  changes  to  the  current  one. 


Below  are  several  aspects  of  the  automated  OJT  system  as 
described  on  the  previous  page.  For  each,  indicate  what  you  feel 
its  value  would  be  to  you  as  an  OJT  manager.  Use  the  following 
scale  for  questions  26-33: 


NO  MODERATE  EXTREME 

VALUE  ^  VALUE  VALUE 

26.  Training  histories  on  individual  airmen 

27.  The  more  task-level  information  than  provided  in  current  STS 

28.  The  standardized  training  practices 

29.  The  standardized  evaluation  procedures 

30.  The  automatic  documentation  of  training 

31.  Providing  training  on  some  tasks  via  the  computer 

32.  Providing  evaluation  on  some  tasks  via  the  computer 

33.  The  automatic  scheduling  of  training 


After  carefully  considering  how  the  system  described  on  the  pre¬ 
vious  page  could  affect  what  you  do  as  an  OJT  manager,  indicate 
how  you  might  change  the  time  you  spend  on  each  training  ac¬ 
tivity.  If,  for  example,  you  believe  you  would  spend  a  little 
more  time  "managing  the  CDC  program"  than  you  do  now,  then  you 


could 

select  option 

g,  "increase  20%. 

It 

a. 

no  change 

<3- 

increase 

20% 

b. 

decrease  100% 

(no  longer  done) 

h. 

increase 

40% 

c. 

decrease  80% 

i. 

increase 

60% 

d. 

decrease  60% 

j  • 

increase 

80% 

e. 

decrease  40% 

k. 

increase 

100%  (double) 

f. 

decrease  20% 

1. 

more  than  doubled 

34.  Managing  upgrade  training  program 
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35.  Conducting/Reporting  staff  assistance  visits 

36 .  Managing  the  CDC  program 

37.  Coordinating  with  CBPO  and  outside  agency  functions 

38.  Assisting  supervisors/commanders 

39.  Organizing/Conducting  meetings 

40.  Daily  supervisory  activities 

41.  Administrative  activities 

Figure  H.8-2 
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PROTOTYPE  AOTS  BASELINE  SURVEY 
SUPERVISORS,  EVALUATORS,  TRAINERS,  AND  TRAINEES 


PRIVACY  ACT  STATEMENT 


PrYIIqi'paL  PURPois!:01'The  SSAN^Ji^^Se^wl^to  identify  personnel  participating  in  the 
R0UTlNE9ui^:em  P'i°tToy5ltennine  the  effectiveness  of  the  current  OJT  program. 


Advanced  On-the-job 


DISCLOSURE  IS  HAtfifATC^Y 'The' jSSAN ,  is  mandatory  to  make  P???^ve  fOenmjMtjoa  °t  i^Mdt^ls  jn  relatiw 
to  other  pe/soryiel  included  in  the  AOJS  brotQtvpe.  tcilure.  to  -  this  lntonnation  wm  Prevent 

required  monitoring  of  training  related  items  wnicn  wilt  jeopardize  the  ability  of  the  AOTS  p.ogram  to  per 
form  its  mission.  . 


DIRECTIONS:  On  the  accompanying  answer  sheet,  please  enter  your  name  (in  the  "Name 
Grid"  block)  and  SSAN  (in  the  "Numeric  Grid"  block,  columns  1  through  9).  Indicate  your 
response  to  each  question  by  filling  in  the  corresponding  oval  for  each  question. 


Note:  Computer  use  does  not  include  the  use  of  a  hand  calculator. 


1.  Have  you  ever  used  a  computer? 
a  -  Yes  b  -  No 


IF  YOU  ANSWERED  NO.  GO  TO  QUESTION  NUMBER  7. 


2.  Do  you  use  a  computer  in  your  job? 

a  -  Yes  b  -  No 

3 .  Have  you  ever  owned  a  computer? 

a  -  Yes  b  -  No 

4.  How  comfortable  do  you  feel  using  a  computer? 

a  -  Very  comfortable 
b  -  Comfortable 

c  -  Neither  comfortable  nor  uncomfortable 
d  -  Uncomfortable 
e  -  Very  uncomfortable 

5.  Have  you  ever  received  instruction  where  the  information  was 
presented  on  a  computer? 

a  -  Yes 
b  -  No 

6.  How  would  you  rate  your  computer-based  learning  experience? 

a  -  N/A,  haven't  received  any  instruction  through  a  computer 
b  -  Outstanding 
e  -•  Excellent 
d  -  Satisfactory 
e  -  Marginal 
f  -  Unsatisfactory 
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7.  Have  you  ever  taken  any  computer  classes? 
a  -  Yes 
b  -  No 


8.  How  often  do  you  play  video  games? 
a  -  Daily 
b  -  Weekly 
c  -  Monthly 
d  -  Yearly 
e  -  Never 


9. 


10. 


What  is  your  rank? 

a. 

AB 

d. 

SrA 

b. 

Amn 

e. 

Sgt 

c. 

A1C 

f . 

SSgt 

or 

above 

How 

long  have  you  been 

in  the 

service? 

a. 

Less  than  3  months 

e. 

12  but 

less 

than 

18 

months 

b. 

3  but  less  than  6  : 

months 

f . 

18  but 

less 

than 

24 

months 

c. 

6  but  less  than  9  : 

months 

g- 

24  but 

less 

than 

36 

months 

d. 

9  but  less  than  2  2 

months 

h. 

36  months  or  more 

11. 


How  long  have  you  been  at  your 

a.  Less  than  3  months 

b.  3  but  less  than  6  months 

c.  6  but  less  than  9  months 

d.  9  but  less  than  12  months 


duty  station? 

e.  12  but  less  than  18 
months 

f.  18  but  less  than  24 
months 

g.  24  but  less  than  36 
months 

h.  36  months  or  more 


12.  What  is  youj-  current  training  status? 

a.  Not  in  training  (already  upgraded/position  qualified) 

b.  Am  certified  in  my  job,  but  have  not  completed  CDCs 

c.  Have  completed  CDCs,  but  not  certified  in  my  job 
d-  In  training  for  both  my  job  and  CDCs 


What 

is/was  tne 

rank  of 

your  trainer? 

a. 

Amn- Ale 

e. 

MSgt 

b. 

SrA-Sgt 

f. 

SMSgt 

c. 

SSgt 

g- 

CMSgt 

d. 

TSgt 

h. 

Civilian 
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14.  Which  of  the  following  best  describes  the  OJT  you  most 
recently  received? 

a.  Active  one-on-one  instruction  by  a  trainer 

b.  Learning  by  doing  (no  trainer  involved) 

c.  Classroom  instruction 

d.  Reading  printed  materials 

e  Computer  based  instruction 

f .  Other 


15 .  What 
your  job? 

a. 

b. 

c. 

d. 

e. 

f. 

g- 

h. 


is/was  your  primary  source  of  information  for  learning 

OJT  trainer 
Co-workers 
Other  persons 

Regulations  and  technical  orders 
Historical  documents  and  records  (files) 

In-shop  training  materials 

Trial  and  error 

Other 


16.  How  are/were  you  principally  evaluated  on  tasks  prior  to 
being  certified  (signed  off)  on  them  in  your  OJT  record?  (Choose 
one. ) 

a.  Trainer  asks/asked  if  I  did  the  task 

b.  Detailed  questioning  by  trainer  (verbal  quiz) 

c.  Written  tests 

d.  Over-the-shoulder  observation  by  trainer  during  entire 
task  performance 

e.  Over-the-shoulder  observation  by  trainer  at  points  in 
task  performance 

f.  Trainer  review  of  completed  work 

g.  Do  not  know 

h.  Other 


17.  From  the  time  you  entered  OJT,  how  long  was  it  before  you 
were  certified  (signed-off  in  your  OJT  record)  on  the  tasks 
you  perform  in  your  job? 


a. 

N/A,  still  training 

g. 

8  months 

b. 

3  months  or  less 

h. 

9  months 

c. 

4  months 

i. 

10  months 

d. 

5  months 

j. 

11  months 

e. 

6  months 

k. 

12  months 

f . 

7  months 

1. 

13  months  or  more 
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18.  From  the  time  you  entered  OJT,  how  long  did  it  take  you  to 
complete  your  CDCs? 


a. 

N/A,  still  studying 

g- 

8  months 

b. 

3  months  or  less 

h. 

9  months 

c. 

4  months 

i. 

10  months 

d. 

5  months 

j  • 

11  months 

e. 

6  months 

k. 

12  months 

f. 

7  months 

1. 

13  months  or  more 

19.  When  was  the  last  time  your  supervisor  reviewed  your 
training  record  with  you? 

a.  Within  the  last  7  days 

b.  Within  the  last  month 

c.  Within  the  last  3  months 

d.  Within  the  last  6  months 

e.  Within  the  last  year 

f.  Within  the  last  year  and  a  half 

g.  Within  the  last  2  years 

h.  My  supervisor  has  never  reviewed  my  record  with  me 

i.  Not  applicable 

20.  About  how  often  does  your  supervisor  review  your  training 
record  with  you? 

a.  Daily 

b .  Weekly 

c .  Monthly 

d.  Quarterly 

e.  Semi-annually 

f .  Annually 

g .  Never 

h.  Not  applicable 

21.  How  would  you  rate  the  overall  quality  of  OJT  you  have 
received? 

a.  Outstanding 

b.  Excellent 

c.  Satisfactory 

d.  Marginal 

e.  Unsatisfactory 


Use  the  scale  below  to  indicate  whether  you  agree 
or  disagree  with  the  statements  that  follow, 

STRONGLY  NEITHER  AGREE  STRONGLY 

DISAGREE  NOR  DISAGREE  AGREE 

a - b - c - d - e - f - g 
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_  22.  My  OJT  program  is/was  well  planned  and  organized. 

_  23.  My  supervisor  is/was  concerned  that  I  receive  good  OJT. 

_  24.  My  OJT  made  me  better  able  to  accomplish  my  job. 

_  25.  My  OJT  trainer  does/did  a  good  job  of  training  me. 

_  26.  I  am  certified  on  tasks  that  I  cannot  really  perform. 

_  27.  More  time  should  have  been  devoted  to  my  OJT. 

_  28.  OJT  is  considered  important  in  my  unit. 

29.  Please  indicate  which  of  the  following  jobs  in  the  OJT 
process  you  have  performed  within  the  past  year:  (answer  all  that 
apply) 


a.  Supervisor 

b.  Evaluator 

c.  Trainer 

d.  Trainee 

e.  All  of  the  above 

f.  None  of  the  above 


IF  YOU  HAVE  BEEN  A  SUPERVISOR,  EVALUATOR,  OR  TRAINER  WITHIN  THE 
PAST  YEAR,  CONTINUE  WITH  QUESTIONS  30  THROUGH  45.  IF  NOT,  STOP  HERE! 


30.  Restrictions  on  deliberately  introducing  problems  on  opera¬ 
tional  equipment  (i.e.,  breaking,  tearing  down,  etc.)  for  train¬ 
ing  purposes  has  the  following  effect  on  my  ability  to  provide 
OJT: 

Adverse  Effect  No  Effect  Positive  Effect 
a  b - c - d - e - f - g 


31.  If  alternative  equipment 
available  for  use  in  training  , 


(simulators,  trainers,  etc.)  were 
the  effect  on  OJT  would  be: 


Adverse  Effect 
a - b - 


c 


No  Effect 
- d - e 


Positive  Effect 
- f - g 
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32.  When  operational  equipment  is  not  readily  available  for 
training,  the  effect  on  OJT  is: 


Adverse  Effect 
a - b - 


No  Effect 


Positive  Effect 

- f - g 


33.  Estimate  the  number  of  times  in  the  past  month  that  you 
would  have  conducted  an  OJT  session,  but  did  not  do  so  because 
operational  equipment  was  not  available. 


a.  0- 

-2  b.  3-5 

c.  6-8 

d.  9 

What 

is  your  rank? 

a. 

Amn 

f . 

TSgt 

b. 

A1C 

g- 

MSgt 

c. 

SrA 

h. 

SMSgt 

d. 

Sgt 

i. 

CMSgt 

e. 

SSgt 

35.  What  is  your  total  time  in  service? 

a.  Less  than  2  years  g. 

b.  2  but  less  than  4  years  h. 

c.  4  out  less  than  6  years  i. 

d.  6  but  less  than  8  years  j. 

e.  8  but  less  than  10  years  k. 

f.  10  but  less  than  12  years 


12  but  less  than  14  years 
14  but  less  than  16  years 
16  but  less  than  18  years 
18  but  less  than  20  years 
20  years  or  more 


36.  In  the  last  year,  how  many  persons  have  you  trained  or 
provided  training  to,  i.e.,  as  their  direct  supervisor  and/or 
trainer? 


a.  None 

b.  l ' 

c.  2 

d.  3 


e.  4 

f.  5 

g.  6 

h.  7  or  more 


37.  How  many  of  the  persons  that  you  directly  supervise  and/or 
train  are  currently  in  upgrade  or  position  qualification 
training? 


a.  None 

b.  l 

c.  2 

d.  3 


e.  4 

f.  5 

g.  6 

h.  7  or  more 
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38.  How  many  tasks  do  you  typically  require  trainees  to  perform 
proficiently  before  being  certified  as  duty  position  qualified? 


a.  1-5 

b.  6-10 

c.  11-15 

d.  16-20 

e.  21-25 

39.  On  an  average,  how  long  does 
become  duty  position  qualified? 

a.  Less  than  3  months 

b.  3  but  less  than  6  months 

c.  6  but  less  than  9  months 

d.  9  but  less  than  12  months 


f.  26-30 

g.  31-35 

h.  36-40 

i.  41-45 

j .  46  or  more 

it  take  one  of  your  trainees  to 


e.  12  but  less  than  18  months 

f.  18  but  less  than  24  months 

g.  24  months  or  more 


40.  Generally,  how  do  you  evaluate  an  airman's  performance  prior 
to  certifying  the  task  in  the  training  record?  (Choose  the  ONE 
method  you  USE  most  often.) 

a.  Ask  the  airman  if  he/she  has  performed  the  task 

b.  Review  the  final  outcome  of  the  task 

c.  Ask  the  airman  to  explain  what  he/she  did  in  performing 
the  task 

d.  Observe  the  airman  perform  the  task  in  the  normal 
course  of  duty 

e.  Check  the  airman's  progress  at  points  throughout  the 
task 

f.  Wait  long  enough  to  insure  the  airman  has  done  the  task 
at  least  once 

g.  Ask  others  who  work  more  closely  with  the  airman  to 
evaluate  his/her  performance 

h.  Set  aside  a  time  for  the  airman  to  perform  the  task 
other  than  in  the  normal  course  of  duty 

i.  Other 

41.  Do  you  document  OJT  for  those  persons  in  job  qualification 
training? 


a.  Yes 

b.  No 

c.  N/A,  supervise  no  one  in  job  qualification  training 
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42.  What  is  the  primary  method  you  use  to  identify  individual 
training  needs?  (Choose  the  ONE  method  you  use  MOST  often.) 

a.  Review  the  airman's  training  record 

b.  Interview  the  airman  and  ask  him/her  what  he/she  has 
done  before 

c.  Ask  the  airman  to  describe  how  he/she  would  perform 
particular  tasks 

d.  Have  the  airman  actually  perform  particular  tasks 

e.  Ask  peers  or  other  supervisors  what  they  think 

f.  Other 


Using  the  following  scale,  indicate  how  important  you  feel  training  is  (in  general,  not  to 
you  particularly)  to  each  of  the  areas  given  below. 


NOT  IMPORTANT  MODERATELY  EXTREMELY 

AT  ALL  IMPORTANT  IMPORTANT 

a  b - c - d - e - f - q 


43. 

Retention  beyond  the 

first  term 

44. 

Initial  decision  to  enter  the  Air  Force 

45. 

Job  satisfaction 

46. 

Safety 

47. 

Morale/Motivation 

48. 

Accomplishment  of  the 

mission 

Below  are  several  specific  training-related  functions  or  activities  you  may  engage  in  as  a 
supervisor  or  trainer.  Review  the  definitions  for  each  function  before  completing  questions 


PROVIDE  TRAINING/SELECT  TRAINERS;  Activities  associated  with 
directly  providing  task  knowledge  and/or  task  proficiency  train- 
lng  for  both  upgrade  and  qualification  trainees.  Also  include 
time  spent  selecting  qualified  trainers  to  provide  training  on 
your  behalf. 
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EVALUATE  TRAINING:  Evaluating  task  knowledge  and/or  task 
proficiency  for  certification  or  recertification  purposes  for 
both  upgrade  and  qualification  trainees.  This  can  be  for  your 
subordinate  personnel  or  as  a  third  party  evaluator  of  other  per¬ 
sonnel  . 

MANAGING  CAREER  DEVELOPMENT  COURSES  (CDCs);  Ensuring  timely 
progression  and  completion  of  CDC  volumes  including  Volume  Review 
Exercises  and  Course  Examinations  and  the  review  training 
required  for  each. 

DEVELOP  TRAINING  PROGRAM  Developing  Master  Training  Plans,  task 
breakdowns,  checklists,  training  schedules,  determining  training 
methods,  advising  trainers  of  correct  training  procedures,  etc. 

COORDINATE  WITH  UNIT  OJT  MANAGER:  Activities  conducted  with  or 
through  the  Unit  OJT  Manager  concerning  OJT  program  procedures, 
documentation  assistance,  initiation  of  upgrade  or  other 
training/personnel  actions,  scheduling  of  training  events,  par¬ 
ticipation  in  formal  or  informal  OJT  visits,  etc. 

COUNSEL  TRAINEES  Counseling  personnel  experiencing  training 
problems. 

DOCUMENT  TRAINING  Documentation  of  AF  Forms  623  and  all  related 
documents  for  upgrade  or  qualification  trainees  and  fully 
qualified  personnel,  maintenance/updating  of  MMICS/ CAMS/ AI CARS  or 
other  automated  training  system  products,  completing  Training 
Quality  Reports  (AF  Form  1284),  and  miscellaneous  training  docu¬ 
ments  and  forms. 

USE  THE  PERCENTAGE  SCALE  BELOW  TO  ANSWER  QUESTIONS  49-56. 


a. 

zero 

f. 

25g. 

30 

k. 

60 

b. 

5 

g- 

35h. 

35 

1. 

70 

c. 

10 

h. 

35i. 

40 

m. 

80 

d. 

15 

i. 

40 j  . 

50 

n. 

90 

e. 

20 

j  • 

50k. 

60 

o. 

100 

49.  On  the  average,  what  percent  of  your  on-the-job  time  do  you 
spend  in  training-related  activities? 

Consider  your  answer  to  question  49  as  a  block  of  time.  Now  in¬ 
dicate  how  this  time  is  distributed  among  each  activity  listed 
below.  Your  answers  will  be  the  relative  percent  of  your  train¬ 
ing  time  you  spend  on  each  activity.  The  total  of  questions  50-56 
should  equal  100%.  It  may  be  helpful  to  first  write  in  your 
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numerical  percentages  directly  on  the  survey  in  the  spaces 
provided.  Then,  using  the  scale  above,  enter  the  appropriate 
letter  response  on  your  answer  sheet.  Be  sure  your  responses 
equal  100%  Refer  to  the  previous  page  for  definitions  of  each 
activity. 

50.  Providing  training  or  selecting  trainers  _ 

51.  Evaluating  performance  _ 

52.  Managing  CDCs  _ 

53.  Developing  training  programs  _ 

54.  Coordinating  with  OJT  Managers  _ 

55.  Counseling  trainees  _ 

56.  Documenting  training  _ 


THIS  IS  A  DESCRIPTION  OF  AN  AUTOMATED  TRAINING  SYSTEM.  Consider 
the  system  to  be  fully  operational.  No  knowledge  of  computer 
programming  or  computer  languages  are  necessary  to  use  the  sys¬ 
tem.  Delivery  is  over  existing  computer  terminals.  This  system 
is  used  with  other  computer  systems  in  the  Air  Force,  like  those 
within  personnel  or  maintenance. 

POSITION-LEVEL  TASK  INFORMATION:  The  focus  of  the  training  sys¬ 
tem  is  on  the  specific  tasks  performed  in  each  position  in  a 
workcenter  or  shop.  Along  with  a  description  of  each  task  are 
given  the  knowledges  and  skills  needed  to  perform  the  task,  the 
standard  for  successful  performance,  a  suggested  training  method, 
and  a  recommended  order  for  training  the  tasks.  In  addition, 
other  types  of  training  requirements  are  identifiec  .  For  ex¬ 
ample,  ancillary  training,  CDCs,  or  training  for  additional 
duties. 

COMPUTERIZED  TRAINING  RECORD:  A  training  record  on  each  airman 
is  kept  on  file  in  the  computer  and  contains  a  list  of  what  the 
airman  has  been  trained  on,  the  training  method  used,  the  date 
the  training  was  completed,  and  the  scores  for  any  tests  or  per¬ 
formance  checks.  Supervisors,  commanders  and  OJT  managers  can 
review  these  training  records  at  any  time.  They  use  the  up-to- 
date  information  from  both  the  airman's  training  record,  and  the 
position  task  requirements  to  identify  where  additional  training 
may  be  needed. 
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AUTOMATIC  SCHEDULING  AND  UPDATING;  Once  an  airman's  training 
requirements  are  identified,  a  training  plan  is  automatically- 
generated  and  he/she  is  scheduled  for  the  training.  Portions  of 
the  training  are  available  to  be,  conducted  "on  line",  that  is  the 
trainee  follows  lessons  and  takes  tests  directly  on  the  computer. 
Computer  programs  exist  within  the  system  to  assist  supervisors 
in  developing  more  localized  training  lessons  and  evaluation 
materials  should  they  be  necessary.  When  a  lesson  is  completed, 
the  date  and  test  results  are  automatically  recorded  in  the 
airman's  computerized  training  record.  Other  tasks  are  trained 
"off-line",  or  away  from  the  computer.  In  this  case,  supervisors 
record  their  evaluations  of  trainee  performance  on  special 
computer-read  forms  so  the  information  can  be  automatically  en¬ 
tered  into  the  airman's  record. 

THIRD-PARTY  EVALUATIONS :  Sometimes  an  airman's  performance  on 
tasks  is  evaluated  by  persons  other  than  his/her  trainer.  These 
"third  party"  evaluations  are  scheduled  automatically,  depending 
on  several  things,  like  the  importance  or  difficulty  of  the  task. 
The  dates  and  results  of  these  performance  checks  are  automati¬ 
cally  entered  into  the  individual  training  record  using  computer- 
read  forms  like  those  for  other  "off-line"  evaluations. 

STANDARDIZED  TRAINING  PRACTICES:  This  automated  training  system 
standardizes  many  aspects  of  the  training  process.  Whenever 
tasks  are  the  same  (on  one  base  or  between  bases)  the  methods 
used  to  train  individuals,  evaluate  their  proficiency,  and  docu¬ 
ment  the  results  should  be  the  same.  The  "third  party"  evalua¬ 
tions  also  help  insure  trainees  are  performing  tasks  ap¬ 
propriately. 

OTHER  USES  OF  THE  TRAINING  INFORMATION:  The  training  information 
stored  in  the  computer  system  is  used  in  several  additional  ways. 
For  example,  managers  can  identify  the  tasks  for  a  specific  posi¬ 
tion  and  then  identify  all  the  persons  who  are  currently 
qualified  to  perform  the  tasks.  This  information  can  be  used  to 
help  select  the  best  person  for  a  job  or  TDY,  select  persons  for 
specialized  training  or  to  attend  PME  courses,  and  for  reporting 
unit  readiness.  In  addition,  this  information  can  help  uncover 
areas  where  many  people  need  training  leading  to  a  new  training 
program  or  changes  to  the  current  one. 

Beiow  are  several  aspects  of  the  automated  OJT  system  as  described  on  the  previous  page. 
For  each,  indicate  what  you  feel  its  value  would  be  to  you  as  a  Supervisor.  Use  the  follow¬ 
ing  scale: 


NO 

MODERATE 

EXTREME 

VALUE 

VALUE 

VALUE 

a - b 

- c - d - e  — 

- —  f - g 
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57.  Training  histories  on  individual  airmen 

58.  The  more  task-level  information  than  provided  in  current  STS 

59.  The  standardized  training  practices 

60.  The  standardized  evaluation  procedures 

61.  The  automatic  documentation  of  training 

62.  Providing  training  on  some  tasks  via  the  computer 

63.  Providing  evaluation  on  some  tasks  via  the  computer 

64.  The  automatic  scheduling  of  training 


After  carefully  considering  how  the  system  described  on  the  previous  page  could  affect 
what  you  do  as  a  Supervisor,  indicate  how  you  might  change  the  time  you  spend  on  each 
training  activity.  If,  for  example,  you  believe  you  would  spend  a  little  more  time  "managing 
the  CDC  program"  than  you  do  now,  then  you  could  select  option  g,  "increase  20%." 


a. 

no  change 

g- 

increase  20% 

b. 

decrease  100% 

(no  longer  done)  h. 

increase  40% 

c. 

decrease  80% 

i. 

increase  60% 

d. 

decrease  60% 

j- 

increase  80% 

e. 

decrease  40% 

k. 

increase  100%  (double) 

f . 

decrease  20% 

1. 

more  than  doubled 

65.  Providing  training  or  selecting  trainers 

66.  Evaluating  performance 

67.  Managing  CDCs 

68.  Developing  training  programs 

69.  Coordinating  with  OJT  Managers 

70.  Counseling  trainees 

71.  Documenting  training 


Figure  H.8-3 
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APPENDIX  I 


DATA  BASE  MANAGEMENT  SYSTEM  ( DBMS ) 


This  Appendix  describes  the  structure  of  the  DBMS  for 
baseline  data  that  can  be  expanded  to  store  System  Level  Test  and 
Evaluation  (SLT&E)  data.  These  data  structures  will  be  used  to 
develop  the  appropriate  code  within  the  ORACLE  data  base  develop¬ 
ment  environment. 

1.1  DATA  BASE  ADMINISTRATION 

A  tightly  administered  DBMS  will  protect  the  data  base  from 
mishandling  and  data  loss.  In  order  to  maintain  the  integrity 
and  validity  of  the  data  collected  during  the  baseline  acquisi¬ 
tion  period,  the  following  data  base  administration  plan  is 
recommended. 

1.1.1  Data  Base  Manager.  The  DAC  Test  Manager  (DTM)  will  act  as 
the  data  base  manager  of  the  Baseline  Data  Base  Management  Sys¬ 
tem.  To  insure  the  integrity  of  the  system,  the  DTM  will  have 
final  responsibility  for  the  loading  and  physical  safety  of  the 
system  (hardware,  software,  and  data) .  The  number  of  personnel 
involved  in  data  entry  tasks  should  be  kept  to  a  minimum.  This 
will  allow  for  less  user  entry  error  and  provide  the  DTM  with  a 
tighter  accounting  of  the  source  of  data  entry  errors. 

1.1.2  Data  Base  Administrator  (DBA) .  Ball  Systems  Engineering 
will  provide  a  DBA.  The  DBA  will  be  responsible  for  maintaining 
the  data  base  system.  The  DBMS  is  designed  in  ORACLE  commercial 
data  base  management  software.  ORACLE  data  base  maintenance 
requires  an  individual  familiar  with  the  intricate  workings  of 
ORACLE  and  Structured  Query  Language  programming.  The  main¬ 
tenance  will  include  allocating  data  storage  and  memory  to  data 
tables,  modifying  code  structures  as  required,  and  designing  any 
new  table  systems. 

1.1.3  Data  Base  Backup  System.  The  backup  system  for  the  data 
base  management  system  will  be  a  tape  backup  system.  The  tape 
backup  system  to  be  procured  should  allow  the  DTM  to  run  a  daily 
backup  of  entered  data.  This  tape  backup  procedure  will  take  ap¬ 
proximately  2-3  minutes.  The  backup  technique  includes  a  rota¬ 
tion  system  such  that  three  backup  tapes  are  used  sequentially. 
If  there  is  ever  a  problem  with  a  tape,  the  next  tape  behind  in 
the  sequence  will  contain  all  but  the  data  backed  up  the  previous 
day. 
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1.1.4  Quality  Control  System.  The  DTM  is  responsible  for  set¬ 
ting  up  a  quality  control  system  including  steps  for  validating 
data  in  the  computerized  data  bases.  The  data  base  screens  will 
be  designed  to  be  near  replications  of  the  data  recording  forms 
from  which  data  is  entered.  Following  are  examples  of  activities 
that  the  DTM  can  institute  in  the  data  base  quality  control 
program: 


Check  each  data  recording  form  with  the  data  entry  screen 
after  data  from  each  data  recording  form  has  been  entered 
into  the  system. 

Retrieve  3%  of  the  records  in  each  data  base  each  quarter 
and  compare  them  to  their  original  data  recording  forms. 


1.2  DATA  BASE  MANAGEMENT  SYSTEM  REPORTS 

The  DBMS  is  designed  with  the  capability  to  produce  user 
generated  ad  hoc  reports.  The  ad  hoc  reporting  system  will  allow 
system  users  to  query  the  data  base  for  information  based  on  any 
conditions.  This  capability  will  be  primarily  used  by  the  DAC 
Test  Manager  in  performing  "Quick-Look  Analysis"  on  the  data.  At 
its  inception,  the  DBMS  will  not  provide  any  standard  reports. 
As  needs  are  defined,  the  Data  Base  Administrator  will  design 
into  the  system  any  standard  reports  desired  by  the  DAC  Test 
Manager. 


1.3  DATA  BASE  MANAGEMENT  SYSTEM  COMPONENTS 

The  overall  design  of  the  DBMS  includes  five  data  table 
systems.  Below  are  the  following  systems  and  the  specific  tables 
included  within  those  systems. 

1)  DESIRE  Listing  Data  Table  System 

DESIRE  ID  Table 
DESIRE  Demographic  Table 

2)  Weekly  Inventory  of  Time  Spent  (WITS)  Sheets  Data  Table 
System 

Unit  Commander  WITS  Table 

Unit  Training  Manager  WITS  Table 

Supervisor/Evaluator/Trainer/Trainee  WITS  Table 

3)  Air  Force  Training  Records  Data  Table  System 

Security  Police  Training  Records  Table 
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Jet  Engine  Training  Records  Table 
Aircraft  Maintenance  Training  Records  Table 
Personnel  Training  Records  Table 

4)  Quality  Assurance  Data  Table  System 

BAFB  Active  Quality  Assurance  Summary  Table 
BAFB  Reserve  Quality  Assurance  Summary  Table 
EANGB  Quality  Assurance  Summary  Table 
Security  Police  Quality  Control  Summary  Table 

5)  Repeat  Maintenance  Action  Data  Table  System 

MDC/MILAP  480  Table 

6)  Base  Crime  Statistics  Data  Table  System 

Base  Crime  Analysis  Report  Table 

7)  Survey  Data  Table  System 

Commander  Baseline  Survey  Table 
OJT  Manager  Baseline  Survey  Table 
Supervisor,  Evaluator,  Trainee,  and  Trainee 
Baseline  Survey  Table 


1.4  DATA  BASE  MANAGEMENT  SYSTEM  STRUCTURAL  DESIGN 

The  Baseline  DBMS  is  designed  using  ORACLE  commercial  data 
base  design  software.  ORACLE'S  design  capabilities  are  based  on 
the  IBM  standard  Structured  Query  Language  (SQL) .  Data  bases 
designed  using  SQL  are  organized  into  data  storage  tables.  The 
logical  design  of  the  SQL  table  is  in  rows  and  columns.  The 
columns  of  the  table  represent  the  data  fields;  the  rows  of  the 
table  represent  the  records  in  the  table.  The  overall  DBMS 
design  is  broken  into  table  systems  that  store  data  of  a  par¬ 
ticular  type. 

I*4*1  DESIRE  Listing  Data.  DESIRE  listing  data  will  be  stored 
in  two  SQL  tables.  The  first  table  will  store  data  from  the 
DESIRE  listing  relative  to  the  specific  identification  of  a  par¬ 
ticular  individual-  Data  from  this  first  table  will  be  utilized 
in  linking  records  on  specific  individuals  in  other  data  tables. 
The  second  table  will  store  some  additional  identification  data 
and  a  great  deal  of  demographic  data  relative  to  that  individual. 
The  two  tables  relating  to  an  individual  subject  will  be  linked 
using  the  Social  Security  Number  field.  The  detailed  table 
designs  follow. 
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DESIRE  ID  TABLE  (Table  Personnel) 


Field  Name 

Width 

Tvoe 

Description 

SSAN 

9 

CHAR  NOT  NULL 

Subject's  Social  Sec.  No. 

RANK 

2 

CHARACTER 

Subject's  Rank 

WORK_CTR 

8 

CHARACTER 

Subject's  Workcenter 

Highest  Total 

Bytes/Record 

19 

Total  Estimated  Number 

of  Records 

525 

Highest  Total 

Overall 

Bytes 

9,975 

DESIRE  DEMOGRAPHIC  TABLE  (TABLE  DESIRE) 

Field  Name 

Width 

Tvoe 

Description 

SSAN 

9 

CHAR  NOT  NULL 

Subject's  Social  Sec.  No. 

LNAME 

27 

CHARACTER 

Subject's  Last  Name 

RECSTAT 

2 

CHARACTER 

Record  Status  &  Condition 

FAC 

6 

CHARACTER 

Functional  Account  Code 

DPN 

7 

CHARACTER 

Duty  Position  Number 

ASVAB  ELD 

2 

CHARACTER 

ELD  ASVAB  Score 

ASVAB  ELE 

2 

CHARACTER 

ELE  ASVAB  Score 

ASVAB  ELF 

2 

CHARACTER 

ELF  ASVAB  Score 

ASVAB  ELG 

2 

CHARACTER 

ELG  ASVAB  Score 

PAFSC 

7 

CHARACTER 

Primary  AFSC 

AFSC2 

7 

CHARACTER 

Secondary  AFSC 

AFSC3 

7 

CHARACTER 

Tertiary  AFSC 

AFSC4 

7 

CHARACTER 

Fourth  AFSC 

CAFSC 

7 

CHARACTER 

Control  AFSC 

DAFSr 

7 

CHARACTER 

Duty  AFSC 

TSTA1 

1 

CHARACTER 

Training  Status  Code 

DTENT  C  W 

6 

CHARACTER 

Date  Entered  Training 

DTENTRT 

6 

CHARACTER 

Date  Entered  Retraining 

EDLEVEL 

10 

CHARACTER 

Education  Level 

RGL  EKM 

3 

CHARACTER 

EKM  Reading  Grade  Level 

RGL  EKN 

4 

CHARACTER 

EKN  Reading  Grade  Level 

RGL  EKP 

3 

CHARACTER 

EKP  Reading  Grade  Level 

RGL  EKQ 

4 

CHARACTER 

EKQ  Reading  Grade  Level 

TEST_TIME 

1 

CHARACTER 

Test  Group 

Highest  Total 

Bytes/Record 

139 

Total  Estimated  Number  of  Records 

525 

Hiyhest  Total 

Overall 

Bytes 

72,975 

The  DESIRE  system  (both  linked  tables)  will  be  used  to  link 
all  other  tables  in  the  DBMS  that  store  data  relative  to  a 
specific  identifiable  individual.  The  Social  Security  field  will 
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be  used  to  provide  this  link.  This  will  allow  for  storing  only 
the  Social  Security  Number  of  an  individual  in  any  records  rela¬ 
tive  to  that  individual  and  providing  additional  personal  infor¬ 
mation  from  the  DESIRE  system.  Details  on  the  specific  linked 
systems  and  the  mechanics  of  the  particular  links  will  be 
provided  in  the  discussions  relative  to  those  systems. 

1.4.2  Weekly  Inventory  Of  Time  Spent  (WITS)  Sheet_Data.  Weekly 
Inventory  Of  Time  Spent  (WITS)  Sheet  data  will  be  stored  in  three 
different  SQL  tables.  There  are  three  different  types  of  WITS 
sheets  that  will  be  administered.  There  is  a  table  for  storing 
each  type  of  existing  WITS  sheet.  The  detailed  designs  of  those 
WITS  data  tables  follow. 


UNIT  COMMANDER  WITS  TABLE  (TABLE  CMDR_WITS) 


Field  Name 

Width 

Type 

Description 

SSAN 

9 

CHARACTER 

Subject's  Social  Sec. 

WK  ENDING 

9 

DATE 

Last  Day  of  Test  Week 

CATEG  ONE 

5 

NUMBER 

Total  of  Category  #1 

CATEG  TWO 

5 

NUMBER 

Total  of  Category  #2 

CAT  THREE 

5 

NUMBER 

Total  of  Category  #3 

CATEG  FOUR 

5 

NUMBER 

Total  of  Category  #4 

CATEG  FIVE 

5 

NUMBER 

Total  of  Category  #5 

CATEG  SIX 

5 

NUMBER 

Total  of  Category  #6 

CATEG  SEVEN 

5 

NUMBER 

Total  of  Category  #7 

CATEG  EIGHT 

5 

NUMBER 

Total  of  Category  #8 

CATEG  NINE 

5 

NUMBER 

Total  of  Category  #9 

TEST_TIME 

1 

CHARACTER 

Test  Group 

Highest  Total 

Bytes/Record 

64 

Total  Estimated  Number  of  Records 

170 

Highest  Total 

Overall 

Bytes 

10,880 

UNIT  TRAINING 

MANAGER 

WITS  TABLE 

(TABLE  TNG_MGR_WITS) 

Field  Name 

Width 

Tvpe 

Description 

SSAN 

£ 

CHARACTER 

Subject's  Social  Sec. 

WK  ENDING 

9 

DATE 

Last  Day  of  Test  Week 

CATEG  ONE 

5 

NUMBER 

Total  of  Category  #1 

CATEG  TWO 

5 

NUMBER 

Total  of  Category  #2 

CATEG  THREE 

5 

NUMBER 

Total  of  Category  #3 

CATEG  FOUR 

5 

NUMBER 

Total  of  Category  #4 

CATEG  FIVE 

5 

NUMBER 

Total  of  Category  #5 

CATEG  SIX 

5 

NUMBER 

Total  of  Category  #6 

CATEG  SEVEN 

5 

NUMBER 

Total  of  Category  #7 
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CATEG_EIGHT  5 

TEST  TIME  1 


NUMBER  Total  of  Category  #8 

CHARACTER  Test  Group 


Highest  Total  Bytes/Record  59 
Total  Estimated  Number  of  Records  216 
Highest  Total  Overall  Bytes  12,744 


SUPERVISOR/EVALUATOR/TRAINER/TRAINEE  WITS  TABLE  (TABLE 
SUP_EVAL_TRAINEE_WITS ) 

Field  Name  Width  Type  Description 


SSAN 

9 

CHARACTER 

Subject's  Social  Sec. 

WK  ENDING 

9 

DATE 

Last 

Day 

of  Test  Week 

CATEG  ONE 

5 

NUMBER 

Total 

of 

Category 

#1 

CATEG  TWO 

5 

NUMBER 

Total 

of 

Category 

#2 

CATEG  THREE 

5 

NUMBER 

Total 

of 

Category 

#3 

CATEG  FOUR 

5 

NUMBER 

Total 

of 

Category 

#4 

CATEG  FIVE 

5 

NUMBER 

Total 

of 

Category 

#5 

CATEG  SIX 

5 

NUMBER 

Total 

of 

Category 

#6 

CATEG  SEVEN 

5 

NUMBER 

Total 

of 

Category 

#7 

CATEG  EIGHT 

5 

NUMBER 

Total 

of 

Category 

#8 

CATEG  NINE 

5 

NUMBER 

Total 

of 

Category 

#9 

CATEG  TEN 

5 

NUMBER 

Total 

of 

Category 

#10 

CATEG  ELEVEN 

5 

NUMBER 

Total 

of 

Category 

#11 

CATEG  TWELVE 

5 

NUMBER 

Total 

of 

Category 

#12 

CATEG_THIRTEEN 

5 

NUMBER 

Total 

of 

Category 

#13 

CATEG  FOURTEEN 

5 

NUMBER 

Total 

of 

Category 

#14 

CATEG  FIFTEEN 

5 

NUMBER 

Total 

of 

Category 

#15 

TEST  TIME 

1 

CHARACTER 

Test  i 

Sroup 

Highest  Total  Bytes/Record 

94 

Total  Estimated 

Number 

of  Records 

6 ,  ‘ 

710 

Highest  Total  Overall 

Bytes 

630,' 

740 

•*••4*3  Air — Force _ Training  Records  Data.  Air  Force  training 

records  are  collected  for  all  test  AFSs  and  each  of  the  three  Air 
Force  components.  The  Air  Force  Training  Records  system  will 
consist  of  four  SQL  data  tables.  The  design  of  the  table  system 
wuc  uauic  lui  tictCn  test  Hi-b.  r,acn  tacie  win  score  aaca 
relative  to  the  individual  test  subject's  training  record.  The 
detailed  table  definitions  for  the  AF  Training  Records  system 
follow. 
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SECURITY  POLICE 

:  TRAINING 

RECORDS  TABLE 

(Table:  SP_TRAINING_REC) 

Field  Name 

Width 

Ty.p_e 

Description 

SSAN 

9 

CHAR  NOT  NULL 

Subject's  Social  Sec.  No 

START  DATE 

9 

DATE 

Date  Assigned  Workcenter 

QUAL  DATE 

9 

DATE 

Date  Position  Qualified 

TSKJENI 

3 

NUMBER 

Tasks  during  initial 
collection 

TSK  MID 

3 

NUMBER 

Tasks  during  mid  coll. 

TSK  FNIj 

3 

NUMBER 

Tasks  during  final  coll. 

OP_OCT87 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Oct/87 

CL_0CT87 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Oct/87 

OP__NOV87 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Nov/87 

CL_NOV87 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Nov/87 

OP_DEC87 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Dec/87 

CL  DEC87 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Dec/87 

OP_JAN88 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Jan/88 

CL _ JAN88 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Jan/88 

OP_FEB88 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Feb/88 

CL_FEB88 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Feb/88 

0P_MAR8  8 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Mar/ 8 8 

CL_MAR8 8 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Mar/88 

OP_APR88 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Apr/ 8 8 

CL  APR88 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Apr/88 

0P_MAY88 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  May/88 

CL_MAY88 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  May/88 

OP_JUN88 

3 

NUMBER 

Total  no t  of  tasks 
opened  by  Jun/88 

CL _ JUN88 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Jun/88 

OP _ 0 UL8 8 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Jul/88 

1-7 


CL_JUL88 

3 

NUMBER 

Total  no.  of  tasks 

COMPONENT 

3 

NUMBER 

closed  by  Jul/88 

Active,  Reserve,  or  ANG 

TEST_TIME 

1 

CHARACTER 

identifier 

Test  Group 

Highest  Total 

Bytes/Record 

100 

Total  Estimated  Number 

of  Records 

45 

Highest  Total 

Overall 

Bytes 

4,500 

JET  ENGINE  TRAINING  RECORDS  TABLE 

(Table:  JET_TRAIING_REC) 

Field  Name 

Width 

Type 

Description 

SSAN 

9 

CHAR  NOT 

NULL  Subject's  Social  Sec.  No. 

START  DATE 

9 

DATE 

Date  Assigned  Workcenter 

QUAL  DATE 

9 

DATE 

Date  Position  Qualified 

TSK_INI 

3 

NUMBER 

Tasks  during  initial 

TSK  MID 

3 

NUMBER 

collection 

Tasks  during  mid  coll. 

TSK  FNL 

3 

NUMBER 

Tasks  during  final  coll. 

OP_OCT87 

3 

NUMBER 

Total  no.  of  tasks 

CL__0CT8  7 

3 

NUMBER 

opened  by  Oct/87 

Total  no.  of  tasks 

OP_NOV87 

3 

NUMBER 

closed  by  Oct/87 

Total  no.  of  tasks 

CL_NOV87 

3 

NUMBER 

opened  by  Nov/87 

Total  no.  of  tasks 

OP_DEC87 

3 

NUMBER 

closed  by  Nov/87 

Total  no.  of  tasks 

CL_DEC87 

3 

NUMBER 

opened  by  Dec/87 

Total  no.  of  tasks 

OP_JAN88 

3 

NUMBER 

closed  by  Dec/87 

Total  no.  of  tasks 

CL_JAN88 

3 

NUMBER 

opened  by  Jan/88 

Total  no.  of  tasks 

OP_FEB88 

3 

NUMBER 

closed  by  Jan/88 

Total  no.  of  tasks 

CL_FEB88 

3 

NUMBER 

opened  by  Feb/88 

Total  no.  of  tasks 

OP_MAR88 

3 

NUMBER 

closed  by  Feb/88 

Total  no.  of  tasks 

CL_MAR8 8 

3 

NUMBER 

opened  by  Mar/ 8 8 

Total  no.  of  tasks 

OP_APR88 

3 

NUMBER 

closed  by  Mar/88 

Total  no.  of  tasks 

CL_APR8  8 

3 

NUMBER 

opened  by  Apr/88 

Total  no.  of  tasks 

closed  by  Apr/88 
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OP_MAY38 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  May/88 

CL_MAY88 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  May/88 

OP_JUN88 

3 

.NUMBER 

Total  no.  of  tasks 
opened  by  Jun/88 

CL_JUN88 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Jun/88 

0P_JUL8  8 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Jul/88 

CL_JUL88 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Jul/88 

COMPONENT 

3 

NUMBER 

Active,  Reserve,  or  ANG 
identifier 

TEST_TIME 

1 

CHARACTER 

Test  Group 

Highest  Total 

Bytes/Record 

100 

Total  Estimated  Number 

of  Records 

116 

Highest  Total 

Overall 

Bytes 

11,600 

AIRCRAFT  MAINTENANCE  TRAINING 
AC_MAINT_TRAINING_REC ) 

RECORDS  TABLE  (Table 

Field  Name 

Width 

Type 

Description 

SSAN 

9 

CHAR  NOT  NULL  Subject's  Social  Sec.  No 

START  DATE 

9 

DATE 

Date  Assigned  Workcenter 

QUAL  DATE 

9 

DATE 

Date  Position  Qualified 

TSK_INI 

3 

NUMBER 

Tasks  during  initial 
collection 

TSK  MID 

3 

NUMBER 

Tasks  during  mid  coll. 

TSK  FNL 

3 

NUMBER 

Tasks  during  final  coll. 

OP_OCT87 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Oct/87 

CL_0CT87 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Oct/87 

OP_NOV87 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Nov/87 

CL_NOV87 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Nov/87 

OP__DEC87 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Dec/87 

CL_DEC87 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Dec/87 

OP_JAN88 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Jan/88 

CL_JAN88 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Jan/88 
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OPJFEB88 

3 

NUMBER 

Total  no. 

of  tasks 

CL_FEB88 

3 

NUMBER 

opened  by 
Total  no. 

Feb/ 8 8 
of  tasks 

OP_MAR88 

3 

NUMBER 

closed  by 
Total  no. 

Feb/ 8 8 
of  tasks 

CL_MAR88 

3 

NUMBER 

opened  by 
Total  no. 

Mar/88 
of  tasks 

OP_APR88 

3 

NUMBER 

closed  by 
Total  no. 

Mar/88 
of  tasks 

CL_APR88 

3 

NUMBER 

opened  by 
Total  no. 

Apr/ 8 8 
of  tasks 

OP_MAY88 

3 

NUMBER 

closed  by 
Total  no. 

Apr/ 8 8 
of  tasks 

CL_MAY88 

3 

NUMBER 

opened  by 
Total  no. 

May/ 8 8 
of  tasks 

OP_JUN88 

3 

NUMBER 

closed  by 
Total  no. 

May/ 8 8 
of  tasks 

CL_JUN88 

3 

NUMBER 

opened  by 
Total  no. 

Jun/88 
of  tasks 

OP_JUL88 

3 

NUMBER 

closed  by 
Total  no. 

Jun/88 
of  tasks 

CL_JUL88 

3 

NUMBER 

opened  by 
Total  no. 

Jul/88 
of  tasks 

COMPONENT 

3 

NUMBER 

closed  by  Jul/88 
Active,  Reserve,  or 

TEST_TIME 

1 

CHARACTER 

identifier 

Test  Group 

Highest  Total  Bytes/Record  100 
Total  Estimated  Number  of  Records  191 
Highest  Total  Overall  Bytes  19,100 


PERSONNEL  TRAINING  RECORDS  TABLE  (Table:  PERS_TRAINING_REC) 


Field  Name 

Width 

Type 

Description 

SSAN 

9 

CHAR  NOT  NULL 

Subject's  Social  Sec.  No. 

START  DATE 

9 

DATE 

Date  Assigned  Workcenter 

QUAL  DATE 

9 

DATE 

Date  Position  Qualified 

TSK_INI 

3 

NUMBER 

Tasks  during  initial 
collection 

TSK  MID 

3 

NUMBER 

Tasks  during  mid  coll. 

TSK_FNL 

3 

NUMBER 

Tasks  dur-inrr  fin^l  coll. 

GP_GCT87 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Oct/87 

CL_OCT8  7 

3 

NUMBER 

Total  no.  of  tasks 
closed  by  Oct/87 

OP_NOV87 

3 

NUMBER 

Total  no.  of  tasks 
opened  by  Nov/87 
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CL_N0V87 

3 

NUMBER 

Total  no. 
closed  by 

of  tasks 
Nov/ 8 7 

0P_DEC87 

3 

NUMBER 

Total  no. 
opened  by 

of  tasks 
Dec/87 

CL_DEC87 

3 

NUMBER 

Total  no. 
closed  by 

of  tasks 
Dec/87 

OP_JAN88 

3 

NUMBER 

Total  no. 
opened  by 

of  tasks 
Jan/88 

CL_JAN88 

3 

NUMBER 

Total  no. 
closed  by 

of  tasks 
Jan/88 

OP_FEB88 

3 

NUMBER 

Total  no. 
opened  by 

of  tasks 
Feb/ 8 8 

CL_FEB88 

3 

NUMBER 

Total  no. 
closed  by 

of  tasks 
Feb/ 8 8 

0P_MAR8  8 

3 

NUMBER 

Total  no. 
opened  by 

of  tasks 
Mar/ 8 8 

CL_MAR88 

3 

NUMBER 

Total  no. 
closed  by 

of  tasks 
Mar/88 

OP_APR88 

3 

NUMBER 

Total  no. 
opened  by 

of  tasks 
Apr/ 8 8 

CL_APR88 

3 

NUMBER 

Total  no. 
closed  by 

of  tasks 
Apr/ 8 8 

OP_MAY88 

3 

NUMBER 

Total  no. 
opened  by 

of  tasks 
May/ 8 8 

CL_MAY88 

3 

NUMBER 

Total  no. 
closed  by 

of  tasks 
May/88 

OP_JUN88 

3 

NUMBER 

Total  no. 
opened  by 

of  tasks 
Jun/88 

CL_JUN88 

3 

NUMBER 

Total  no. 
closed  by 

of  tasks 
Jun/88 

0P_JUL8£ 

3 

NUMBER 

Total  no. 
opened  by 

of  tasks 
Jul/88 

CL_JUL88 

3 

NUMBER 

Total  no. 
closed  by 

of  tasks 
Jul/88 

COMPONENT 

3 

NUMBER 

Active,  Reserve,  or 
identifier 

TESTJTIME 

1 

CHARACTER 

Test  Group 

Highest  Total 

Bytes/Record 

100 

Total  Estimated  Number 

of  Records 

30 

Highest  Total 

Overall 

Bytes 

3,000 

Quality  Assurance  Data .  Quality  Assurance  reports  will  be 
collected  from  all  components  at  Bergstrom  Air  Force  Base  and  El¬ 
lington  Air  National  Guard  Base.  The  Quality  Assurance  Data  Sys¬ 
tem  will  consist  of  four  SQL  data  tables.  The  first  data  table 
will  contain  the  data  collected  from  the  Active  QA  Monthly  Sum¬ 
maries. 
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QUALITY  ASSURANCE  ACTIVE  SUMMARY  TABLE  (Table:  QA_ACTIVE_  SUM) 


Field  Name 

Width 

Type 

Description 

QUARTER 

1 

NUMBER 

Quarter  of  report  (l,  2, 

3,  or  4) 

AMUJE 

3 

NUMBER 

AMU  Excellent  Technical 
Inspections  (TIs) 

AMU  S 

3 

NUMBER 

AMU  Satisfactory  TIs 

AMU  U 

3 

NUMBER 

AMU  Unsatisfactory  TIs 

AMU  TOT 

3 

NUMBER 

AMU  Total  TIs 

AMU_SAT_PE 

3 

NUMBER 

AMU  Satisfactory  Per¬ 
sonnel  Evaluations 

AMU_UN_PE 

3 

NUMBER 

AMU  Unsatisfactory 
Personnel  Evaluations 

AMU_TOT_PE 

3 

NUMBER 

AMU  Total  Personnel 
Evaluations 

PROP  E 

3 

NUMBER 

Propulsion  Excellent  TIs 

PROP_S 

3 

NUMBER 

Propulsion  Satisfactory 
TIs 

PROP_U 

3 

NUMBER 

Propulsion  Unsatisfactory 
TIs 

PROP  TOT 

3 

NUMBER 

Propulsion  Total  TIs 

PROP_SAT_PE 

3 

NUMBER 

Propulsion  Satisfactory 
Personnel  Evaluations 

PRO  P_UN_PE 

3 

NUMBER 

Propulsion  Unsatisfactory 
Personnel 

PROP_TOT_PE 

3 

NUMBER 

Propulsion  Total 

Personnel  Evaluations 

PM  E 

3 

NUMBER 

Phase  Maint. Excellent  TIs 

PM_S 

3 

NUMBER 

Phase  Maint.  Satisfactory 
TIs 

PM_U 

3 

NUMBER 

Phase  Maintentance 
Unsatisfactory  TIs 

PM  TOT 

3 

NUMBER 

Phase  Maint.  Total  TIs 

PM_SAT_PE 

NUMBER 

Phase  Maint.  Satisfactory 
Personnel  Evaluations 

PM_TOT_PE 

3 

NUMBER 

Phase  Maint.  Unsatisfact. 
Personnel  Evaluations 

PM_TOT_PE 

3 

NUMBER 

Phase  Maint.  Total 
Personnel  Evaluations 

TEST_TIME 

1 

CHARACTER 

Test:  Group 

Highest  Total 

Bytes/Record 

65 

Total  Estimated  Number 

of  Records 

10 

Highest  Total 

Overall 

Bytes 

650 
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QUALITY  ASSURANCE  RESERVE  SUMMARY  TABLE  (Table:  QA_RESERVE_SUM) 


Field  Name 

Width 

Tvpe 

Description 

QUARTER 

1 

NUMBER 

Quarter  of  report  (1,  2 
3 ,  or  4 ) 

AMB_TOT 

3 

NUMBER 

AMB  Total  Technical 
Inspections  (TIs) 

AMB  E 

3 

NUMBER 

AMB  Excellent  TIs 

AMB  S 

3 

NUMBER 

AMB  Satisfactory  TIs 

AMB  U 

3 

NUMBER 

AMB  Unsatisfactory  TIs 

PROP  TOT 

3 

NUMBER 

Propulsion  Total  TIs 

PROP_E 

3 

NUMBER 

Propulsion  Excellent 

TIs 

PROP_S 

3 

NUMBER 

Propulsion  Satisfactory 
TIs 

PROP_U 

3 

NUMBER 

Propulsion  Unsatis.  TIs 

PE  TOT 

3 

NUMBER 

Total  Personnel  Evalua. 

PE  PASS 

3 

NUMBER 

Passed  Personnel  Evalua 

PE  FAIL 

3 

NUMBER 

Failed  Personnel  Evalua 

TEST_TIME 

1 

CHARACTER 

Test  Group 

Highest  Total 

Bytes/Record 

35 

Total  Estimated  Number 

of  Records 

10 

Highest  Total 

Overall 

Bytes 

350 

QUALITY  ASSURANCE  AIR  NATIONAL  GUARD  SUMMARY  TABLE 
( Table :  QA_RESERVE_SUM) 


Field  Name 

Width 

Type 

Description 

QUARTER 

1 

NUMBER 

Quarter  of  report  (1,  2, 

3,  or  4) 

AMU_T0T 

3 

NUMBER 

AMU  Total  Technical 
Inspections  (TIs) 

AMU  SAT 

3 

NUMBER 

AMU  Satisfactory  TIs 

AMU  UNSAT 

3 

NUMBER 

AMU  Unsatis.  TIs 

AMU_T0T_PE 

3 

NUMBER 

AMU  Total  Personnel 
Evaluations 

AMU_SAT_PE 

3 

NUMBER 

AMU  Satisfactory 

Personnel  Evaluations 

AMU_UNSAT_PE 

3 

NUMBER 

AMU  Unsatisfactory 
Personnel  Evaluations 

PROP  TOT 

3 

NUMBER 

Propulsion  Total  TIs 

PROP  SAT 

3 

NUMBER 

Propulsion  Satis.  TIs 

PROP_UNSAT 

3 

NUMBER 

Propulsion  Unsatisfactory 
TIs 

PRO  P_TOT_PE 

3 

NUMBER 

Propulsion  Total 

Personnel  Evaluations 
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PRO  P_S AT_PE  3 
PROP_UNSAT_PE  3 
TEST  TIME  1 


NUMBER 

NUMBER 

CHARACTER 


Propulsion  Satisfactory 
Personnel  Evaluations 
Propulsion  Unsatisfactory 
Personnel  Evaluations 
Test  Group 


Highest  Total  Bytes/Record 
Total  Estimated  Number  of  Records 
Highest  Total  Overall  Bytes 


38 

10 

380 


SECURITY  POLICE  QUALITY  CONTROL  DATA  TABLE  (Table:  SP_QC_LETTER) 


Field  Name 

Width 

Tvoe 

Description 

QUARTER 

1 

NUMBER 

Quarter 
3,  or  4) 

of  report  (1, 

2, 

SPA 

3 

NUMBER 

Average 

score. 

Flight 

SPA 

SPO 

3 

NUMBER 

Average 

score, 

Flight 

SPO 

SPOL_A 

3 

NUMBER 

Average 

SPOL-A 

score, 

Flight 

SPOL_B 

3 

NUMBER 

Average 

SPOL-B 

score, 

Flight 

SPOL_C 

3 

NUMBER 

Average 

SPOL-C 

score, 

Flight 

SPOL_D 

3 

NUMBER 

Average 

SPOL-D 

score, 

Flight 

TEST_TIME 

1 

CHARACTER 

Test  Group 

Highest  Total  3ytes/Record  20 
Total  Estimated  Number  of  Records  4 
Highest  Total  Overall  Bytes  80 


1.4.5  Repeat  Maintenance  Action  Data.  Data  relative  to  the  num¬ 
ber  of  repeat  maintenance  actions  occurring  on  aircraft  assigned 
to  the  maintenance  test  workcenters  is  collected  from  the  MDC  and 
MILAP  480  reports.  The  MDC  and  MILAP  480  data  will  be  stored  in 
one  SQL  data  table.  The  detailed  table  definitions  for  the 
Repeat  Maintenance  Action  Data  system  follow. 


MDC/MILAP  480  REPORT  TABLE  (Table 


Field  Name 

Width 

Type 

MDC  DATE 

9 

DATE 

PROPRPTS 

2 

NUMBER 

SP  FLIGHT-RPTS 

3 

NUMBER 

ACMNTRPTS 

3 

NUMBER 

MDC_MILAP_RPT) 

Description 

Date  of  Report 
Propulsion  Repeat  Actions 
Specialist  Flight  Repeats 
Aircraft  MX  Repeat 
Actions 


PHASERPTS 

MILAP_A 

MILAP_B 

SME 

TEST  TIME 


2 

3 

3 

15 

1 


NUMBER 

NUMBER 

NUMBER 

CHARACTER 

CHARACTER 


Phase  Repeat  Actions 
Repeat  Actions  on  MILAP 
A  Flight 

Repeat  Actions  on  MILAP 
B  Flight 

SME  Reviewing  Report 
Test  Group 


Highest  Total  Bytes/Record 
Total  Estimated  Number  of  Records 
Highest  Total  Overall  Bytes 


41 

10 

410 


1*4.6  Base  Crime  Statistics  Data.  Base  crime  statistics  will  be 
collected  at  BAFB  from  the  Monthly  Crime  Analysis  Report.  infor¬ 
mation  from  this  report  will  be  stored  in  one  SQL  data  storage 
table.  The  detailed  design  of  the  table  follows. 


BASE  CRIME  ANALYSIS  REPORT  (Table 


Field  Name 

Width 

Type 

SURVEY  DATE 

9 

DATE 

CR  PERS 

3 

NUMBER 

CR  PROP 

3 

NUMBER 

CR_GOV_PROP 

3 

NUMBER 

CR_PRIV_PROP 

3 

NUMBER 

CR  DRUG  INC 

3 

NUMBER 

ADMIN  ERROR 

3 

NUMBER 

TEST  TIME 

1 

CHARACTER 

Highest  Total  Bytes/Record 
Total  Estimated  Number  of  Records 
Highest  Total  Overall  Bytes 


SP_CRIME_ANALY  SIS) 

Description 
Report  Date 

#  Crimes  Against  Persons 

#  Crimes  Against  Property 

#  Crimes  Against  Gov't. 
Property 

#  Crimes  Against  Private 
Property 

#  of  Drug  Incidents 

#  Administrative  Errors 
Test  Group 

28 

4 

112 


I*4*7  Survey  Data.  The  results  of  all  surveys  administered 
during  the  baseline  data  acquisition  period  will  be  stored  in  the 
Survey  Data  System.  There  will  be  three  types  of  surveys  ad¬ 
ministered.  The  data  collected  from  these  surveys  will  be  stored 
m  three  data  tables  corresponding  to  each  type  of  survey. 

I. 4. 7.1  Baseline  Commander  Survey  Data.  The  results  of  the 
Baseline  Commander  Surveys  will  be  stored  in  the  Baseline  Com¬ 
mander  Survey  Table.  This  single  table  will  store  all  of  the 
identifying  data  and  responses  taken  from  the  surveys.  The 
Baseline  Commander  Survey  table  will  be  linked  to  the  DESIRE  sys- 
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tem  through  the  Social  Security  Number  field.  This  will  allow 
detailed  information  regarding  the  individual  survey  respondent 
to  be  accessed.  The  detailed  design  of  the  table  follows. 


BASELINE  COMMANDER  SURVEY  TABLE  (Table: 


Field 

Name 

Width 

Typo 

Description 

NAME 

27 

CHARACTER 

Name  of  Respondent 

SSAN 

9 

CHARACTER 

Subjects 

Social  Sec.  No. 

QUEST 

1 

1 

CHARACTER 

Answer 

to 

Question 

#1 

QUEST 

"2 

1 

CHARACTER 

Answer 

to 

Question 

n 

QUEST 

"3 

1 

CHARACTER 

Answer 

to 

Question 

#3 

QUEST 

"4 

1 

CHARACTER 

Answer 

to 

Question 

#4 

QUEST- 

'5 

1 

CHARACTER 

Answer 

to 

Question 

#5 

QUEST 

"6 

1 

CHARACTER 

Answer 

to 

Question 

#6 

QUEST" 

1 

CHARACTER 

Answer 

to 

Question 

#7 

QUEST- 

"8 

1 

CHARACTER 

Answer 

to 

Question 

#8 

QUEST 

"9 

1 

CHARACTER 

Answer 

to 

Question 

#9 

QUEST- 

"10 

1 

CHARACTER 

Answer 

to 

Question 

#10 

QUEST 

"ll 

1 

CHARACTER 

Answer 

to 

Question 

#11 

QUEST 

12 

1 

CHARACTER 

Answer 

to 

Question 

#12 

QUEST 

13 

1 

CHARACTER 

Answer 

to 

Question 

#13 

QUEST 

14 

1 

CHARACTER 

Answer 

to 

Question 

#14 

QUEST- 

15 

1 

CHARACTER 

Answer 

to 

Question 

#15 

QUEST 

16 

1 

CHARACTER 

Answer 

to 

Question 

#16 

QUEST 

17 

1 

CHARACTER 

Answer 

to 

Question 

#17 

QUEST 

18 

1 

CHARACTER 

Answer 

to 

Question 

#18 

QUEST 

19 

1 

CHARACTER 

Answer 

to 

Question 

#19 

QUEST 

"20 

7 

CHARACTER 

Answer 

to 

Question 

#20 

QUEST- 

"21 

1 

CHARACTER 

Answer 

to 

Question 

#21 

QUEST 

22 

1 

CHARACTER 

Answer 

to 

Question 

#22 

QUEST- 

"23 

1 

CHARACTER 

Answer 

to 

Question 

#23 

QUEST 

"24 

1 

CHARACTER 

Answer 

to 

Question 

#24 

QUEST 

"25 

1 

CHARACTER 

Answer 

to 

Question 

#25 

QUEST 

"26 

1 

CHARACTER 

Answer 

to 

Question 

#26 

QUEST- 

27 

1 

CHARACTER 

Answer 

to 

Question 

#27 

QUEST 

"28 

1 

CHARACTER 

Answer 

to 

Question 

#28 

quest" 

"29 

1 

CHARACTER 

Answer 

to 

Question 

#29 

quest" 

30 

1 

CHARACTER 

Answer 

to 

Question 

#30 

quest" 

"3.1 

1 

CHARACTER 

Answer 

to 

Question 

#31 

quest" 

"3  2 

1 

CHARACTER 

Answer 

to 

Question 

#32 

quest" 

'33 

1 

CHARACTER 

Answer 

to 

Question 

#33 

QUEST 

-34 

240 

CHARACTER 

Answer 

to 

Question 

#34 

quest" 

~35 

240 

CHARACTER 

Answer 

to 

Question 

#35 

test_time 

1 

CHARACTER 

Test  Group 

Highest  Total  Bytes/Record  556 
Total  Estimated  Number  of  Records  525 
Highest  Total  Overal  Bytes  291,900 
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1 . 4 . 7 . 2  Baseline  OJT  Manager  Survey  Data.  A  Baseline  OJT 
Manager  Survey  will  be  administered  to  all  OJT  and  Unit  Training 
Managers  in  the  test  workcenters.  The  data  collected  from  these 
surveys  will  be  stored  in  one  SQL  data  table.  The  detailed 
design  of  this  table  follows. 


BASELINE  OJT  MANAGER  SURVEY  TABLE  (Table: 


Field 

Name 

Width 

Tyee 

Description 

NAME 

27 

CHARACTER 

Name  of  Respondent 

SSAN 

9 

CHARACTER 

Subject's 

Social  Sec.  : 

QUEST 

1 

1 

CHARACTER 

Answer 

to 

Question 

#1 

quest" 

‘2 

1 

CHARACTER 

Answer 

to 

Question 

#2 

quest" 

"3 

1 

CHARACTER 

Answer 

to 

Question 

#3 

quest" 

~4 

1 

CHARACTER 

Answer 

to 

Question 

#4 

quest" 

"5 

1 

CHARACTER 

Answer 

to 

Question 

#5 

quest" 

"6 

1 

CHARACTER 

Answer 

to 

Question 

#6 

quest" 

"7 

1 

CHARACTER 

Answer 

to 

Question 

#7 

quest" 

'8 

1 

CHARACTER 

Answer 

to 

Question 

#8 

quest’ 

'9 

1 

CHARACTER 

Answer 

to 

Question 

#9 

quest" 

'lO 

1 

CHARACTER 

Answer 

to 

Question 

#10 

quest" 

11 

1 

CHARACTER 

Answer 

to 

Question 

#11 

quest 

12 

1 

CHARACTER 

Answer 

to 

Question 

#12 

quest" 

13 

1 

CHARACTER 

Answer 

to 

Question 

#13 

quest" 

14 

1 

CHARACTER 

Answer 

to 

Question 

#14 

quest 

15 

1 

CHARACTER 

Answer 

to 

Question 

#15 

quest 

16 

1 

CHARACTER 

Answer 

to 

Question 

#16 

quest 

17 

1 

CHARACTER 

Answer 

to 

Question 

#17 

quest 

~13 

1 

CHARACTER 

Answer 

to 

Question 

#18 

quest 

19 

1 

CHARACTER 

Answer 

to 

Quest j  on 

#19 

QUEST 

*20 

7 

CHARACTER 

Answer 

to 

Question 

#20 

QUEST 

*21 

1 

CHARACTER 

Answer 

to 

Question 

#21 

QUEST 

22 

1 

CHARACTER 

Answer 

to 

Question 

#22 

QUEST 

~23 

1 

CHARACTER 

Answer 

to 

Question 

#23 

QUEST 

’24 

1 

CHARACTER 

Answer 

to 

Question 

#24 

quest" 

25 

1 

CHARACTER 

Answer 

to 

Question 

#25 

quest- 

~2  6 

1 

CHARACTER 

Answer 

to 

Question 

#26 

quest" 

‘27 

1 

CHARACTER 

Answer 

to 

Question 

#27 

QUEST 

'28 

1 

CHARACTER 

Answer 

to 

Question 

#28 

quest" 

29 

1 

CHARACTER 

Answer 

to 

Question 

#29 

quest" 

'30 

1 

CHARACTER 

Answer 

to 

Question 

#30 

quest" 

'31 

1 

CHARACTER 

Answer 

to 

Question 

#31 

quest- 

>2 

1 

CHARACTER 

Answer 

to 

Question 

11  *"*  0 

n->* 

quest” 

33 

I 

CHARACTER 

Answer 

to 

Question 

#33 

quest” 

*34 

240 

CHARACTER 

Answer 

to 

Question 

#34 

quest" 

"35 

240 

CHARACTER 

Answer 

to 

Question 

#35 

quest" 

3  6 

1 

CHARACTER 

Answer 

to 

Question 

#26 

QUEST_ 

’37 

1 

CHARACTER 

Answer 

to 

Question 

#27 

quest" 

"38 

1 

CHARACTER 

Answer 

to 

Question 

#28 
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QUEST_39  1  CHARACTER  Answer  to  Question  #29 
QUEST_40  1  CHARACTER  Answer  to  Question  #30 
QUEST_4 1  1  CHARACTER  Answer  to  Question  #31 
TESTJTIME  1  CHARACTER  Test  Group 

Highest  Total  Bytes/Record  552 
Total  Estimated  Number  of  Records  227 
Highest  Total  Overall  Bytes  125,304 


I . 4 . 7 . 3  Supervisor.  Evaluator.  Trainer,  and  Trainee  Baseline 
Survey  Data.  Supervisor,  Evaluator,  and  Trainer  Baseline  Surveys 
will  be  administered  to  all  of  the  subjects  in  all  of  the  test 
workcenters.  The  data  collected  from  these  surveys  will  be 
stored  in  one  data  table.  The  detailed  design  of  this  table  fol¬ 
lows. 


SUPERVISOR,  EVALUATOR,  TRAINER,  AND  TRAINEE  BASELINE  SURVEY  TABLE 


Field 

Name 

Width 

Type 

Description 

NAME 

27 

CHARACTER 

Name  of  Respondent 

S3AN 

9 

CHARACTER 

Subject's 

Social  Sec.  No 

QUEST 

1 

1 

CHARACTER 

Answer 

to 

Question 

#1 

QUEST 

"2 

1 

CHARACTER 

Answer 

to 

Question 

#2 

QUEST 

”3 

1 

CHARACTER 

Answer 

to 

Question 

#3 

QUEST 

”4 

1 

CHARACTER 

Answer 

to 

Question 

#4 

QUEST 

"5 

1 

CHARACTER 

Answer 

to 

Question 

#5 

quest” 

”6 

1 

CHARACTER 

Answer 

to 

Question 

#6 

quest” 

”7 

3 

CHARACTER 

Answer 

to 

Question 

#7 

quest” 

”8 

1 

CHARACTER 

Answer 

to 

Question 

#8 

quest 

"9 

1 

CHARACTER 

Answer 

to 

Question 

#9 

quest” 

”io 

1 

CHARACTER 

Answer 
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12 

7,104 


1.5  DATA  BASE  LOGICAL  STRUCTURE 


The  logical  structure  of  the  AOTS  Baseline  Data  Base 
Management  System  is  divided  into  two  table  sets: 

Workcenter  Data  Table  Set — Figure  1.6-1,  p.  21 
Test  Subject  Data  Table  Set —  Figure  1.6-2,  p.  22 

The  Test  Subject  set  contains  tables  specifically  related  to 
individual  test  subjects.  In  the  context  of  this  system,  each 
record  in  the  Test  Subject  set  relates  to  one  individual  (i.e. 
WITS  sheets  are  filled  out  by  specific  individuals,  training 
record  data  is  collected  on  specific  individuals) .  Each  subject 
in  the  baseline  data  collection  will  have  a  DESIRE  Listing 
record.  The  DESIRE  Listing  record  will  include  all  of  the  neces¬ 
sary  identifying  information  on  the  individual.  The  DESIRE  List¬ 
ing  record  contains,  among  other  data  elements,  the  individual's 
Social  Security  Number.  All  of  the  files  in  the  Test  Subject  set 
have  the  Social  Security  Number  as  a  data  element.  This  allows 
all  of  the  records  relative  to  an  individual  to  be  linked 
together  and  to  be  linked  back  to  the  DESIRE  Listing  record. 
This  allows  demographic  information  (including  the  individual's 
first  and  last  name,  workcenter,  etc.)  to  be  entered  into  the 
system  only  once. 

The  Workcenter  Test  set  contains  data  tables  that  store  data 
relative  to  the  workcenters.  These  table  records  cannot  be 
tracked  to  a  specific  individual,  only  to  a  particular  workcen¬ 
ter. 
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TEST  SUBJECT  DATA  TABLE  SET 
LOGICAL  TABLE  STRUCTURE 
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APPENDIX  J 


SYSTEM  LEVEL  TEST  AND  EVALUATION  f SLT&E) 
DATA  COLLECTION  PROCEDURES 


This  appendix  describes  methodologies  to  be  used  during 
Phase  III  SLT&E  to  collect  data  from  the  AOTS  experimental  and 
control  groups  at  Bergstrom  AFB  and  Ellington  ANGB  and  from  non¬ 
participant  observers  from  representative  MAJCOMS.  The 
methodologies  include  the  administration  of  several  surveys  and 
the  continued  collection  of  the  Baseline  data,  excluding  the  ad¬ 
ministration  of  the  Prototype  AOTS  Baseline  Survey. 

The  surveys  to  be  administered  during  SLT&E  include: 

Opinion  Surveys  to  be  administered  to  Supervisors,  Training 
Managers,  Trainers  and  Evaluator 
Acceptance  Surveys  to  be  administered  to  Commanders, 
Supervisors,  Training  Managers,  Trainees, 

Trainers  and  Evaluators 

Deployment  Surveys  to  be  administered  to  Commanders, 

Supervisors,  Training  Managers,  Evaluators,  and 
Trainers 

Observer  Surveys  to  be  administered  to  visitors  attending  a 
two  day  AOTS  demonstration 

The  Baseline  data  collections  extended  during  SLT&E  include: 

Test  Subject  Demographic  Data 
OJT  Records  Data 
Quality  Controls  Data 
Aircraft  Maintenance  Data 
Base  Crime  Statistics  Data 
WITS  Sheets  Data 

A  cross  reference  to  the  critical  issues  and  subquestion (s) 
in  Appendix  E  and  pages  for  step-by-step  procedures  to  obtain 
SLT&E  data  are  as  follows: 

SECTION  I:  SLT&E  SURVEYS 

J.l  SLT&E  OPINION  SURVEYS  .  4 

— Mission  Readiness  (E-10B) 

— Trainee  Performance  (E-12) 

— Training  &  Evaluation  Requirements  (E-16A) 

— Trainee  Qualifications  (E-17A) 

— Use  of  AOTS  Products  for  Trainee  Qualifications 
(E-17B) 

— Evaluation  Methods  for  Workcenters  (E-18A) 

— Management  of  Availability  of  Qualified  Trainers 


J-l 


(E-21A) 

— Enhancement  of  Trainer  Effectiveness  (E-22A) 

— Management  of  Training  (E-23A) 

— Instructional  Technology  (E-24A) 

J.2  ACCEPTANCE  SURVEYS  .  20 

— Management  of  Training  in  Workcenters  (E-32A) 

— Management  of  Products  (E-32B) 

— Training  on  Management  Programs  &  Management 
Data  (E-32C) 

— Functions  of  Training  Delivery  Programs  (E-33B) 

— Capabilities  for  Evaluating  Training  (E-34A) 

— Functions  of  Evaluation  Programs  (E-34B) 

— Training  on  Evaluation  Programs  (E-34C) 

— Training  on  Operation  of  Training  Delivery  Pro¬ 
grams  (E-33C) 

— Capabilities  for  Authoring  Training  Materials  (E33A) 


J .  3  AOTS  DEPLOYMENT  INTERVIEW/SURVEY  (E-25A) . 40 

J.4  OBSERVERS  SURVEY  .  53 


— Functions  of  Training  Delivery  Programs  (E-33B) 

— Management  of  Training  in  Workcenters  (E-32A) 

— Capabilities  for  Evaluating  Training  (E-34A) 

— Functions  of  Evaluation  Programs  (E-34B) 

— Training  &  Evaluation  Documents  Comparison  (E-16B) 
— Evaluation  Instruments  &  Procedures  (E-18B) 

— Operational  Equipment  Alternatives  (E-20B) 

— Products  &  Processes  For  Trainer  Effectiveness 
(E-22B) 


SECTION  II:  SLT&E  PROCEDURES  CONTINUED  FROM  BASELINE 


J.  5  TEST  SUBJECT  DEMOGRAPHIC  DATA . 61 

J.  6  OJT  RECORDS  (E-7A,  E-7B)  .  .....  62 

J.  7  QUALITY  ASSURANCE  SUMMARIES  (E-8A)  .  65 

J.  8  SECURITY  POLICE  QUALITY  CONTROL  TREND  (E-8A)  .  66 

J.  9  AIRCRAFT  MAINTENANCE  DATA  (E-9A) . 67 

J.10  BASE  CRIME  STATISTICS  (E-10A)  68 

J.ll  WITS  SHEETS  (E-11A,  E-15A,  E-23B)  69 
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SECTION  III:  SLT&E  COMPLIANCE  PROCEDURES 


J . 12  EXTERNAL  ENTITIES  INTERFACES  (E-5A)  .  70 

J . 13  RESPONSE  PERFORMANCE  (E-5B)  .  76 

J. 14  HARDWARE  RELIABILITY  (E-5C)  .  78 

J.35  HARDWARE  MAINTAINABILITY  (E-5D)  .  82 

J. 16  SYSTEM  AVAILABILITY  (E-5E)  .  89 
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Section  I:  SURVEYS 


J.l  SLT&E  OPINION  SURVEYS 

J.1.1  Data  To  Be  Collected.  The  Opinion  Surveys  are  designed 
to  collect  opinions  from  AOTS  participants  on  whether  AOTS  is 
better,  worse,  or  no  different  from  conventional  OJT.  SLT&E 
Opinion  Surveys  consist  of  data  collected  from  three  sources  as 
follows: 

Supervisors — Figure  J.l-1,  p.  8 
Training  Managers — Figure  J.l-2,  p.  14 
Trainers/Evaluators — Figure  J.l-3,  p.  16 

If  a  person  acts  as  both  a  Supervisor  and  an  Evaluator  or  as 
both  a  Supervisor  and  a  Trainer,  he/she  should  be  categorized  as 
a  Supervisor. 

Using  a  response  scale  ranging  from  A  to  G,  the  respondents 
are  asked  to  indicate  how  strongly  they  oppose  or  favor  the  ef¬ 
fects  of  AOTS.  Areas  covered,  with  Appendix  E  references,  are  as 
follows: 

Mission  Readiness  (E-10B) 

Trainee  Performance  (E-12A) 

Training  &  Evaluation  Requirements  (E-16A) 

Trainee  Qualifications  (E-17A) 

Use  of  AOTS  Products  for  Trainee  Qualifications  (E-17B) 
Evaluation  Methods  for  Workcenters  (E-18A) 

Management  of  Operational  Equipment  (E-19A) 

Management  of  Availability  of  Qualified  Trainers  (E-21A) 
Enhancement  of  Trainer  Effectiveness  (E-22A) 

Management  of  Training  (E-23A) 

Instructional  Technology  (E-24A) 

J*1.2  Requirements  for  DCR.  The  DCR  must  have  the  ability  to 
administer  surveys  individually  or  in  a  group  administration. 

J-1.3  Frequency  of  Data  Collection.  The  Opinion  Surveys  will 
be  administered  one  time  during  the  third  quarter  of  SLT&E  (May 
1989) 

Letters,,  of  Request.  Prior  to  administering  the  surveys, 
the  DCR  will  submit  a  letter  to  the  Squadron  Commander  requesting 
all  AOTS  participants  attend  the  Opinion  Survey  administration 
event.  Squadron  Commanders  and  location  are  found  in  Attachment 
1  to  Appendix  J. 
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The  DCR 


will  administer 


J . 1 . 5  Administration  of  Surveys. 
the  surveys.  The  DCR  will: 

-  Obtain  a  sufficient  number  of  #  2  pencils,  Optical  Mark 
Reader  (OMR)  sheets  and  Opinion  Surveys  (Supervisor, 
Training  Manager,  and  Trainer/Evaluator)  for  distribution 
to  the  scheduled  AOTS  participants. 

-  Attend  and  supervise  the  administration  event. 

—  Administer  the  surveys  in  a  room  free  from  distractions 
—  Distribute  the  surveys  and  OMR  sheets  to  each 
participant 

—  Provide  instructions  for  completing  the  surveys 
(see  sample  instructions  below) 

—  Allow  the  respondents  to  leave  and  return  to  the 
administration  area  without  the  surveys,  (Surveys 
should  remain  with  the  DCR  while  respondents  are  absent 
from  the  survey  administration  area) 

—  Answer  any  questions  pertaining  to  the  completion  of 
the  forms 

—  Review  ALL  OMR  sheets  before  the  respondents  leave 
the  administration  area  and  ensure  that: 

The  correct  SSAN  is  entered 
The  correct  Answer  Sheet  number  is  entered 
All  question  and  statement  responses  are  entered 
—  Collect  all  pencils,  surveys  and  OMR  sheets  after 
the  surveys  have  been  completed 

SUGGESTIONS  FOR  ADMINISTERING  SURVEYS 

-  Administer  more  than  one  type  (Opinion,  Acceptance, 
Deployment)  of  surveys  at  the  same  administration  event 
when  appropriate 

-  Allow  for  errors  (bring  extra  copies  of  surveys  and  OMR 
sheets) 

-  Administer  the  surveys  in  a  classroom,  conference  room  or 
theatre 

-  Apply  a  ratio  of  one  DCR  to  approximately  15  respondents 

-  Administer  surveys  to  personnel  from  multiple  workcenters 
at  a  central  location  when  possible 

-  Read  the  following  SAMPLE  INSTRUCTIONS  to  the  respondents 

THE  PURPOSE  OF  THIS  SURVEY  IS  TO  DETERMINE  HOW  PERSONNEL  IN 
THE  WORKCENTERS,  WHO  HAVE  OPERATED  THE  AOTS,  FEEL  ABOUT  THE  SYS¬ 
TEM.  USING  THE  ACCOMPANYING  OMR  SHEET,  INDEPENDENTLY  RESPOND  TO 
ALL  QUESTIONS  AND  STATEMENTS.  NOTE  THAT  SOME  STATEMENTS  MAY  AP¬ 
PEAR  TO  BE  IDENTICAL  TO  OTHERS  BUT  AFTER  CLOSER  EXAMINATION  YOU 
WILL  SEE  THAT  EACH  STATEMENT  REFERS  TO  A  DIFFERENT  TOPIC.  READ 
EACH  STATEMENT  CAREFULLY  BECAUSE  SOME  STATEMENTS  FAVOR  AOTS  AND 
SOME  FAVOR  CONVENTIONAL  OJT.  AFTER  YOU  HAVE  RESPONDED  TO  ALL  OF 
THE  STRUCTURED  STATEMENTS,  USE  THE  BACK  OF  THE  SURVEY  TO  REPORT 
ANY  COMMENTS  OR  SUGGESTIONS  THAT  YOU  HAVE  ABOUT  AOTS.  BE  HONEST 
AND  TAKE  AS  LONG  AS  YOU  WANT  TO  RESPOND  TO  THE  SURVEY. 
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SAMPLE  LETTER  FOR  SCHEDULING  PERSONNEL  FOR  SLT&E  SURVEYS 


REPLY  TO 

ATTN  OF :  AFHRL/OL-AK 

SUBJECT:  AOTS  Survey  Administration 

TO:  67  AGS/CC 


1.  Request  the  following  personnel  be  scheduled  to  report  to 
Building  T-l  (across  from  Bldg  1604)  at  1300  hrs  on  29  Feb  1988. 
The  individuals  will  be  asked  to  respond  to  surveys  regarding 
their  involvement  with  the  prototype  Advanced  OJT  System  (AOTS) . 
These  appointments  should  take  approximately  1  hour. 


Captain  Hays,  Jeffrey  D. 
CMSgt  Singleton,  Joseph 
SMSgt  Dreher,  Steven  L. 
SMSgt  Gosc ,  Robert  L. 
SMSgt  D'Amico,  James 
MSgt  Elliott,  Thomas  D. 
TSgt  Brooks,  David  W. 
SSgt  Hand,  Betsy 
A1C  Leger,  Lynn 
Amn  West,  James  J. 

AB  Sturdevant,  Wayne  A. 


FR000923847 

FR555223671 

FR84392019S 

FR321664335 

FR631226734 

FR773400812 

FR543889201 

FR211211211 

FR231987885 

FR012994999 

FR460403233 


12th  AMU,  B  Flight 
12th  AMU,  A  Flight 
12th  AMU,  B  Flight 
12th  AMU,  A  Flight 
12th  AMU,  B  Flight 
12th  AMU,  B  Flight 
12th  AMU,  A  Flight 
12th  AMU,  A  Flight 
12th  AMU,  B  Flight 
12th  AMU,  A  Flight 
12th  AMU,  B  Flight 


JACK  L.  BLACKHURST,  Major,  USAF 
Commander,  AFHRL/OL-AK 
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J .  1 . 6  Time  Required  for  Data  Collection . 

1.5  hrs.  per  workcenter — AFHRL  distribute/collect  surveys 

1.0  hrs.  per  workcenter — Input  data  into  AOTS  data  table 

1.0  hr.  per  individual — Complete  surveys 

J.1.7  Disposition  of  Data.  After  the  surveys  have  been  col¬ 
lected,  DCR  will  conduct  a  quick-look  analysis  of  the  results  to 
insure  the  integrity  of  the  data  entered  by  the  respondents.  The 
following  plan  will  be  followed: 

-  Verify  that  all  SSANs  are  entered  and  that  the  SSANs 
correspond  with  the  names  and  SSANs  on  the  list  of 
incumbents . 

-  Verify  that  all  response  marks  on  each  OMR  sheet  are 
within  the  tolerance  limits  for  the  optical  scanner. 

-  Verify  that  each  form's  demographic  data  meets  the  stand¬ 
ards  set  on  the  sample  OMR  sheet  for  the  same  sur¬ 
vey. 

If  records  are  found  to  be  out  of  tolerance  or  with  error, 
the  DCR  will  return  the  OMRs  to  the  respondent  for  immediate  cor¬ 
rection. 

ACCURATE  records  will  be  given  to  the  Data  Input 
Representative  (DIR)  for  data  input. 
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SUPERVISOR  SLT&E  OPINION  SURVEY 


DIRECTIONS :  On  the  accompanying  answer  sheet,  please  enter: 

your  NAME  in  the  ••NAME  GRID"  block, 

your  SSAN  in  columns  1-9  of  the  “NUMERIC  GRID”  block, 
number  04  in  columns  24/25  of  the  “NUMERIC  GRID”  block. 

Indicate  your  response  to  each  statement  by  filling  in  the  oval, 
on  the  answer  sheet,  that  corresponds  to  the  letter  that  repre¬ 
sents  your  answer  or  opinion. 

Read  every  statement  VERY  carefully  before  selecting  the  response 
option  that  matches  your  opinion. 


1.  AS  A  SUPERVISOR,  WHAT  CATEGORY  OF  TRAINEES  HAVE  YOU  BEEN 
OBSERVING  DURING  THE  DEPLOYMENT  OF  THE  AOTS  PROTOTYPE? 

A.  ONLY  TRAINEES  TRAINED  UNDER  AOTS 

B.  TRAINEES  TRAINED  UNDER  AOTS  AND  TRAINEES  TRAINED  UNDER 
CONVENTIONAL  OJT 


Unless  otherwise  instructed,  use  the  following  set  of  response 
options  to  indicate  your  opinions: 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

2.  TRAINEES  ARE  BETTER  ABLE  TO  PERFORM  TASKS  WHEN  TRAINED  UNDER 
AOTS  THAN  WHEN  TRAINED  UNDER  CONVENTIONAL  OJT. 

3.  TRAINEES  ADAPT  MORE  QUICKLY  TO  CHANGING  MISSION  DEMANDS  WHEN 
TRAINED  UNDER  AOTS  THAN  WHEN  TRAINED  UNDER  CONVENTIONAL  OJT. 

4.  PEOPLE  IN  MY  WORKCENTER  MAKE  MORE  MISTAKES  PERFORMING  TASKS 
AFTER  BEING  TRAINED  UNDER  AOTS  THAN  AFTER  BEING  TRAINED  UNDER 
CONVENTIONAL  OJT. 

5.  TRAINEES  ARE  BETTER  ABLE  TO  MEET  MISSION  READINESS 
REQUIREMENTS  AFTER  BEING  TRAINED  UNDER  AOTS  THAN  AFTER  BEING 
TRAINED  UNDER  CONVENTIONAL  OJT. 
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Unless  otherwise  instructed,  use  the  following  set  of  response 
options  to  indicate  your  opinions: 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

6.  TASK  TRAINING  IS  MORE  THOROUGH  UNDER  AOTS  THAN  UNDER 
CONVENTIONAL  OJT. 

7.  TRAINEES  HAVE  A  BETTER  UNDERSTANDING  OF  TASK  PERFORMANCE 
REQUIREMENTS  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

8.  THE  USE  OF  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  HAS 
INCREASED  THE  TRAINEES '  ABILITY  TO  PERFORM  TASKS. 

9.  TRAINEES  REQUIRE  LESS  SUPERVISION  WHEN  PERFORMING  TASKS  AFTER 
BEING  TRAINED  UNDER  AOTS  THAN  AFTER  BEING  TRAINED  UNDER 
CONVENTIONAL  OJT. 

10.  TRAINEES  RETAIN  KNOWLEDGE  AND  SKILLS  LONGER  WHEN  TRAINED 
UNDER  AOTS  THAN  WHEN  TRAINED  UNDER  CONVENTIONAL  OJT. 

11.  AOTS  BEHAVIORAL  OBJECTIVES  ENABLE  ME  TO  DETERMINE  TRAINING 
REQUIREMENTS  BETTER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

12.  AOTS  BEHAVIORAL  OBJECTIVES  ENABLE  ME  TO  IDENTIFY  TASK 
EVALUATION  STANDARDS  BETTER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL 
OJT. 

13.  REMEDIAL  TRAINING  REQUIREMENTS  FOR  TASKS  ARE  MORE  EASILY 
IDENTIFIED  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

14.  I  AM  LESS  ABLE  TO  IDENTIFY  SPECIFIC  TRAINING  NEEDS  WHEN  I 
USE  THE  RESULTS  FROM  THE  AOTS  TASK  PERFORMANCE  EVALUATIONS  THAN 
WHEN  I  USE  THE  RESULTS  FROM  CONVENTIONAL  OJT  EVALUATIONS. 

15.  I  CAN  KEEP  TRACK  OF  A  TRAINEE'S  PROGRESS  TOWARD  TASK 

proficiency  better  using  aotg  procedures  than  using  conventional 

OJT  METHODS. 

16.  AOTS  BEHAVIORAL  OBJECTIVES  CLEARLY  INDICATE  THE  REQUIRED 
STANDARD  OF  TASK  PERFORMANCE. 
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SUPERVISOR  SLT&E  OPINION  SURVEY 


Unless  otherwise  instructed,  use  the  following  set  of  response 
options  to  indicate  your  opinions: 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

17.  I  AM  BETTER  ABLE  TO  DETERMINE  A  TRAINEE'S  QUALIFICATIONS 
USING  TASK  STATEMENTS  WRITTEN  UNDER  AOTS  THAN  STATEMENTS  WRITTEN 
UNDER  CONVENTIONAL  OJT. 

18.  AOTS  PRODUCTS,  SUCH  AS  AIRMAN  TRAINING  RECORDS  (ATRs) ,  ARE 
MORE  HELPFUL  THAN  CONVENTIONAL  OJT  PRODUCTS  IN  DETERMINING 
TRAINEE  QUALIFICATIONS. 

19.  AOTS  EVALUATION  STANDARDS  ARE  TOO  HIGH  FOR  THE  TASKS 
PERFORMED  IN  MY  WORKCENTER. 

20.  AOTS  EVALUATION  INSTRUCTIONS  ARE  CLEAR  AND  EXPLICIT. 

21.  AOTS  TESTS  CONTAIN  ENOUGH  DETAIL  TO  ADEQUATELY  DETERMINE 
TRAINEE  QUALIFICATIONS . 

22.  AOTS  TESTS  COVER  STEPS  AND  PROCEDURES  THAT  DO  NOT  REQUIRE 
EVALUATION. 

23.  QUALIFIED  TRAINERS  ARE  MORE  EASILY  IDENTIFIED  UNDER  AOTS 
THAN  UNDER  CONVENTIONAL  OJT. 

24.  TRAINERS  ARE  ABLE  TO  MANAGE  MORE  TRAINEES  UNDER  AOTS  THAN 
UNDER  CONVENTIONAL  OJT. 

25.  TRAINERS  ARE  ABLE  TO  IDENTIFY  AVAILABLE  TRAINING  MATERIALS 
MORE  READILY  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

26.  TRAINERS  APPEAR  TO  BE  MORE  CONFIDENT  IN  THEIR  TRAINING 
ABILITIES  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

27.  TRAINING  IS  MORE  RELIABLE  UNDER  AOTS  THAN  UNDER  CONVENTIONAL 
OJT. 


28.  MANAGING  THE  DESIGNATION  OF  QUALIFIED  TRAINERS  IS  EASIER 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 
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SUPERVISOR  SLT&E  OPINION  SURVEY 


Unless  otherwise  instructed,  use  the  following  set  of  response 
options  to  indicate  your  opinions: 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

29.  TRAINERS  ARE  ABLE  TO  DO  A  BETTER  JOB  OF  TRAINING  UNDER  AOTS 
THAN  UNDER  CONVENTIONAL  OJT. 

30.  TRAINERS  ARE  ABLE  TO  GIVE  TRAINEES  MORE  INDIVIDUALIZED 
TRAINING  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

31.  TRAINERS  HAVE  MORE  DIFFICULTY  JUDGING  INDIVIDUAL  PROGRESS 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

32.  TRAINERS  ARE  BETTER  ABLE  TO  JUDGE  TRAINEES '  TASK  PROFICIENCY 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

33.  TRAINERS  ARE  ABLE  TO  GET  TRAINEES  POSITION  QUALIFIED  FASTER 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

34.  AOTS  REDUCES  THE  TIME  SPENT  MAINTAINING  TRAINING  RECORDS. 

35.  SCHEDULING  TRAINING  IS  MORE  DIFFICULT  UNDER  AOTS  THAN  UNDER 
CONVENTIONAL  OJT. 


36.  TRACKING  TRAINEES'  PROGRESS  IN  THE  WORKCENTER  IS  EASIER 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

37.  AOTS  ASSISTS  ME  IN  KEEPING  RECORDS  FOR  TRACKING  TRAINEES' 
PROGRESS  TOWARDS  POSITION  QUALIFICATION. 

38.  UPDATING  AOTS  TRAINING  RECORDS  (e.g.,  ATRs,  ITRs)  IS  LESS 
TIME  CONSUMING  THAN  COMPLETING  THE  PAPER  WORK  NEEDED  TO  MANAGE 
TRAINING  UNDER  CONVENTIONAL  OJT. 

39.  AOTS  PRODUCES  REPORTS  THAT  CAN  BE  USED  TO  EVALUATE  TRAINERS' 
COMPETENCE  i.N  TRAINING  WORKCENTER  TASKS. 

40.  I  SPEND  LESS  TIME  COORDINATING  TRAINING  UNDER  AOTS  THAN 
UNDER  CONVENTIONAL  OJT. 
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Unless  otherwise  instructed/  use  the 
options  to  indicate  your  opinions: 

following 

set 

of  response 

A  B 

C  D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

41.  THE  AVAILABILITY  OF  COMPUTER  ASSISTED  INSTRUCTION  (CAI)  IN 
THE  WORKCENTER  IS  GREATER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

42.  THE  CAPABILITY  FOR  COMPUTER  MANAGEMENT  OF  TRAINING  IS 
GREATER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

43.  THE  CAPABILITY  FOR  IDENTIFYING  TRAINING  REQUIREMENTS  IS 
GREATER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

44.  THE  CAPABILITY  FOR  IDENTIFYING  TRAINING  DEFICIENCIES  IS 
GREATER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

45.  DOING  THE  JOB  IN  THE  OPERATIONAL  ENVIRONMENT  IS 

MORE  DIFFICULT  AFTER  BEING  TRAINED  WITH  AOTS  CAI  THAN  WHEN  BEING 
TRAINED  UNDER  CONVENTIONAL  OJT. 

46.  OVERALL,  TRAINING  TIME  DECREASED  AFTER  PERSONNEL  BECAME 
FAMILIAR  WITH  THE  ON-LINE  COMPUTER-ASSISTED  TRAINING  AND 
EVALUATION  FEATURES  OF  THE  AOTS. 
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SUPERVISOR  SLT&E  OPINION  SURVEY 


USE  THE  FOLLOWING  RESPONSE  OPTION  SCALE  TO  RESPOND  TO  STATEMENTS 
47-51. 


A 

B 

C 

D 

E 

F 

G 

NONE  OF 

THE  TIME. 

SOMETIMES. 

ALL  OF 
THE  TIME 

47.  I  USE  AOTS  GENERIC  POSITION  TASK  REQUIREMENTS  (GPTRs)  TO 
DETERMINE  THE  TRAINEE ' S  POSITION  TRAINING  REQUIREMENTS. 

48.  I  USE  THE  AOTS  QUALIFICATION  ASSESSMENT  CAPABILITIES  TO 
DETERMINE  TRAINEE  QUALIFICATIONS 

49.  I  USE  AOTS  GENERATED  REPORTS  TO  DETERMINE  HOW  WELL  TRAINEES 
ARE  PROGRESSING  IN  TRAINING 

50.  I  USE  AOTS  PERFORMANCE  TESTS  (PERFORMANCE  EVALUATION 
CHECKLISTS)  TO  DETERMINE  TRAINEE  QUALIFICATIONS 

51.  I  USE  AOTS  TESTS  TO  DETERMINE  IF  TRAINEES  POSSESS  THE 
KNOWLEDGE  REQUIRED  TO  PERFORM  TASKS 


COMMENTS : 


TRAINING  MANAGER  SLT&E  OPINION  SURVEY 


DIRECTIONS.  On  the  accompanying  answer  sheet,  please  enter: 
your  NAME  in  the  "NAME  GRID'*  block, 

your  SSAN  in  columns  1-9  of  the  "NUMERIC  GRID"  block, 
number  06  in  columns  24/25  of  the  "NUMERIC  GRID"  block. 

Indicate  your  response  to  each  statement  by  filling  in  the  oval, 
on  the  answer  sheet,  that  corresponds  to  the  letter  that  repre¬ 
sents  your  answer  or  opinion. 

Read  every  statement  VERY  carefully  before  selecting  the  response 
option  that  matches  your  opinion. 

Use  the  following  set  of  response  options  to  indicate  your 
opinions: 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

1.  AOTS  BEHAVIORAL  OBJECTIVES  ENABLE  ME  TO  DETERMINE  TRAINING 
REQUIREMENTS  BETTER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

2.  AOTS  BEHAVIORAL  OBJECTIVES  ENABLE  ME  TO  IDENTIFY  TASK  EVALUA¬ 
TION  STANDARDS  BETTER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

3.  REMEDIAL  TRAINING  REQUIREMENTS  FOR  TASKS  ARE  MORE  EASILY 
IDENTIFIED  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

4.  I  AM  LESS  ABLE  TO  IDENTIFY  SPECIFIC  TRAINING  NEEDS  WHEN  I  USE 
THE  RESULTS  FROM  THE  AOTS  TASK  EVALUATIONS  THAN  WHEN  I  USE  THE 
RESULTS  FROM  CONVENTIONAL  OJT  EVALUATIONS. 

5.  I  CAN  KEEP  TRACK  OF  A  TRAINEE'S  PROGRESS  TOWARD  TASK 
PROFICIENCY  BETTER  USING  AOTS  PROCEDURES  THAN  USING  CONVENTIONAL 
OJT  METHODS. 

6.  BY  REVIEWING  AOTS  INDIVIDUAL  TRAINING  REQUIREMENTS  (ITRs) ,  I 
AM  BETTER  ABLE  TO  FOCUS  MY  ATTENTION  ON  POSITION  QUALIFICATION  AS 
OPPOSED  TO  JUST  UPGRADE  TRAINING. 
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TRAINING  MANAGER  SLT&E  OPINION  SURVEY 


Use  the  following  set  of  response  options  to  indicate  your 
opinions: 


A 

B 

C 

D 

E 

P 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

7.  I  AM  BETTER  ABLE  TO  DETERMINE  THE  STATUS  OF  UPGRADE  TRAINING 
IN  THE  WORKCENTERS  BY  REVIEWING  THE  AOTS  WORKCENTER  UPGRADE 
TRAINING  REPORTS  THAN  BY  REVIEWING  CONVENTIONAL  TRAINING  STATUS 
REPORTS . 

8.  TRAINING  NECESSARY  TO  ACHIEVE  TASK  PROFICIENCY  IS  MORE 
EASILY  IDENTIFIED  USING  THE  EVALUATION  MATERIALS  (e.g. ,  KNOWLEDGE 
TESTS,  PERFORMANCE  TESTS)  DEVELOPED  UNDER  AOTS  THAN  EVALUATION 
MATERIALS  DEVELOPED  UNDER  CONVENTIONAL  OJT. 

9.  MANAGING  THE  DATA  REQUIRED  IN  MY  OJT  PROGRAM  IS 
MORE  DIFFICULT  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

10.  BECAUSE  OF  THE  QUALITY  OF  DATA  CONTAINED  IN  THE  AOTS 
REPORTS,  I  AM  BETTER  ABLE  TO  ADVISE  ON  TRAINING  NEEDS  AND 
DEFICIENCIES  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

11.  OVERALL,  TRAINING  TIME  DECREASED  AFTER  TRAINEES  BECAME 
FAMILIAR  WITH  THE  ON-LINE  COMPUTER-ASSISTED  TRAINING  AND 
EVALUATION  FEATURES  OF  THE  AOTS. 


COMMENTS : 


Pirtnvft  T  *1—0 

*•  -’’3  ^  w  ♦  X  £. 


J—  1 5 


TRAINER/EVALUATOR  SLT&E  OPINION  SURVEY 


DIRECTIONS :  On  the  accompanying  answer  sheet,  please  enter: 
your  NAME  in  the  "NAME  GRID”  block, 

your  SSAN  in  columns  1-9  of  the  "NUMERIC  GRID”  block, 
number  05  in  columns  24/25  of  the  "NUMERIC  GRID”  block. 

Using  the  following  set  of  response  options,  indicate  your 
response  to  each  statement  by  filling  in  the  oval,  on  the  answer 
sheet,  chat  corresponds  to  the  letter  that  represents  your  answer 
or  opinion. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

Read  every  statement  VERY  carefully  before  selecting  the  response 
option  that  matches  your  opinion. 


1.  TRAINEES  HAVE  A  BETTER  UNDERSTANDING  OF  TASK  PERFORMANCE 
REQUIREMENTS  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

2.  THE  USE  OF  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  HAS 
INCREASED  THE  TRAINEES'  ABILITY  TO  PERFORM  RELATED  TASKS. 

3.  TRAINEES  REQUIRE  LESS  SUPERVISION  WHEN  PERFORMING  TASKS  AFTER 
BEING  TRAINED  UNDER  AOTS  THAN  AFTER  BEING  TRAINED  UNDER 
CONVENTIONAL  OJT. 


4.  TRAINEES  RETAIN  KNOWLEDGE  AND  SKILLS  LONGER  WHEN  TRAINED 
UNDER  AOTS  THAN  WHEN  TRAINED  UNDER  CONVENTIONAL  OJT. 

5.  TRAINEES  PERFORM  THEIR  TASKS  WITH  FEWER  ERRORS  WHEN  TRAINED 
UNDER  AOTS  THAN  WHEN  TRAINED  UNDER  CONVENTIONAL  OJT. 


6.  AOTS  BEHAVIORAL  OBJECTIVES  ENABLE  ME  TO  DETERMINE  TRAINING 
REQUIREMENTS  BETTER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 


/.  AUTt>  ctnaviUKAjj  OBJECTIVES  ENABLE  ME  TO  IDENTIFY  TASK  EVALUA¬ 


TION  STANDARDS  BETTER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 


8.  REMEDIAL  TRAINING  REQUIREMENTS  FOR  TASKS  ARE  MORE  EASILY 
IDENTIFIED  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 
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Use  the  following  set  of  response  options  to  indicate  your 
opinions: 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

9.  I  AM  LESS  ABLE  TO  IDENTIFY  SPECIFIC  TRAINING  NEEDS  WHEN  I  USE 
THE  RESULTS  FROM  THE  AOTS  TASK  PERFORMANCE  EVALUATIONS  THAN  WHEN 
I  USE  THE  RESULTS  FROM  CONVENTIONAL  OJT  EVALUATIONS. 

10.  I  CAN  KEEP  TRACK  OF  A  TRAINEE'S  PROGRESS  TOWARD  TASK  PRO¬ 
FICIENCY  BETTER  USING  AOTS  PROCEDURES  THAN  USING  CONVENTIONAL  OJT 
METHODS . 

11.  AOTS  EVALUATION  STANDARDS  ARE  TOO  HIGH  FOR  THE  TASKS 
PERFORMED  IN  MY  WORKCENTER. 

12.  AOTS  EVALUATION  INSTRUCTIONS  ARE  CLEAR  AND  EXPLICIT. 

13.  AOTS  TESTS  CONTAIN  ENOUGH  DETAIL  TO  ADEQUATELY  DETERMINE 
TRAINEE  QUALIFICATIONS. 

14.  AOTS  TESTS  COVER  STEPS  AND  PROCEDURES  THAT  DO  NOT  REQUIRE 
EVALUATION. 


15.  QUALIFIED  TRAINERS  ARE  MORE  EASILY  IDENTIFIED  UNDER  AOTS 
THAN  UNDER  CONVENTIONAL  OJT. 

16.  TRAINERS  ARE  ABLE  TO  MANAGE  MORE  TRAINEES  UNDER  CONVENTIONAL 
OJT  THAN  UNDER  AOTS. 


17.  TRAINERS  ARE  ABLE  TO  IDENTIFY  AVAILABLE  TRAINING  MATERIALS 
MORE  READILY  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 


D  D2  MORE  CONFIDENT  IN 
ABILITIES  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 


rTVDTv  TMTMP. 


19.  TRAINING  IS  MORE  RELIABLE  UNDER  AOTS  THAN  UNDER  CONVENTIONAL 
OJT. 


20.  MANAGING  THE  DESIGNATION  OF  QUALIFIED  TRAINERS  IS  EASIER 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 
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Use  the  following  set  of  response  options  to  indicate  your 
opinions: 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

21.  TRAINERS  ARE  ABLE  TO  DO  A  BETTER  JOB  OF  TRAINING  UNDER  AOTS 
THAN  UNDER  CONVENTIONAL  OJT. 

22.  TRAINERS  ARE  ABLE  TO  GIVE  TRAINEES  MORE  INDIVIDUALIZED 
TRAINING  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

23.  TRAINERS  HAVE  MORE  DIFFICULTY  JUDGING  INDIVIDUAL  PROGRESS 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

24.  TRAINERS  ARE  BETTER  ABLE  TO  JUDGE  TRAINEES'  TASK  PROFICIENCY 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

25.  TRAINERS  ARE  ABLE  TO  GET  TRAINEES  POSITION  QUALIFIED  FASTER 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

26.  AOTG  REDUCES  THE  TIME  SPENT  MAINTAINING  TRAINING  MANAGEMENT 
RECORDS . 

27.  SCHEDULING  TRAINING  IS  MORE  DIFFICULT  UNDER  AOTS  THAN  UNDER 
CONVENTIONAL  OJT. 

28.  TRACKING  TRAINEES'  PROGRESS  IN  THE  WORKCENTER  IS  EASIER 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

29.  AOTS  ASSISTS  ME  IN  KEEPING  RECORDS  FOR  TRACKING  TRAINEES' 
PROGRESS  TOWARDS  POSITION  QUALIFICATION. 

30.  UPDATING  AOTS  TRAINING  RECORDS  (e.g. ,  ATRs,  ITRs)  IS  LESS 
TIME  CONSUMING  THAN  COMPLETING  THE  PAPER  WORK  NEEDED  TO  MANAGE 
TRAINING  UNDER  CONVENTIONAL  OJT. 

31.  AOTS  PRODUCES  REPORTS  THAT  CAN  BE  USED  TO  EVALUATE  TRAINERS' 
COMPETENCE  IN  TRAINING  WORKCENTER  TASKS. 

32.  I  SPEND  LESS  TIME  COORDINATING  TRAINING  UNDER  AOTS  THAN 
UNDER  CONVENTIONAL  OJT. 
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Use  the  following  set  of 
opinions: 

response 

options 

to  indicate  your 

ABC 

D 

E 

F  G 

STRONGLY 

NEUTRAL 

STRONGLY 

DISAGREE 

AGREE 

33.  THE  AVAILABILITY  OF  COMPUTER  ASSISTED  INSTRUCTION  (CAI)  IN 
THE  WORKCENTER  IS  GREATER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

34.  THE  CAPABILITY  FOR  COMPUTER  MANAGEMENT  OF  TRAINING  IS 
GREATER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OUT. 

35.  THE  CAPABILITY  OF  IDENTIFYING  TRAINING  REQUIREMENTS  IS 
GREATER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

36.  THE  CAPABILITY  OF  IDENTIFYING  TRAINING  DEFICIENCIES  IS 
GREATER  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

37.  DOING  THE  JOB  IN  THE  WORKCENTER  IS  MORE  DIFFICULT  AFTER 
BEING  TRAINED  WITH  AOTS  CAI  THAN  WHEN  BEING  TRAINED  UNDER 
CONVENTIONAL  OJT. 

38.  OVERALL,  TRAINING  TIME  DECREASED  AFTER  PERSONNEL  BECAME 
FAMILIAR  WITH  THE  ON-LINE  COMPUTER-ASSISTED  TRAINING  AND 
EVALUATION  FEATURES  OF  THE  AOTS. 

COMMENTS : 


Figure  J.l-3 
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J . 2  ACCEPTANCE  SURVEYS 


j„2.1  Data  To  Be  Collected.  The  Acceptance  Surveys  are  designed 
to  collect  views  from  ACTS  participants  on  whether  AGTS  is  easy 
to  operate  and  user  friendly.  Acceptance  Surveys  consist  of  data 
collected  from  six  sources  as  follows: 

Supervisors — Figure  J.2-1,  p.  25 
Training  Managers — Figure  J.2-2,  p.  29 
Trainers/Evaluators — Figure  J.2-3,  p.  33 
Trainees- — Figure  J.2-4,  p.  34 
Commanders — Figure  J.2-5,  p.  36 
Training  Developers — Figure  J.2-6,  p.  38 

Individuals  who  perform  the  function  of  more  than  one  category 
should  be  considered  to  be  in  the  first  of  those  categories 
listed  above.  For  example,  if  a  person  acts  as  both  a  Super¬ 
visor  and  an  Evaluator  or  as  both  a  Supervisor  and  a  Trainer, 
he/ she  should  be  categorized  as  a  Supervisor. 


Using  a  response  scale  ranging  from  A  to  G,  the  respondents  are 
asked  to  indicate  how  acceptable  they  believe  the  AOTS  to  be. 
Areas  covered,  with  Appendix  E  references,  are  as  follows: 

Management  of  Training  in  Workcenters  (E-32A) 

Management  of  Products  (E-32B) 

Training  on  Management  Programs  &  Management  Data  (E-32C) 
Functions  of  Training  Delivery  Programs  (E-33B) 

Capabilities  for  Evaluating  Training  (E-34A) 

Functions  of  Evaluation  Programs  (E-34B) 

Training  on  Evaluation  Programs  (E-34C) 

Training  on  Operation  of  Training  Delivery  Programs  (E-33C) 
Capabilities  for  Authoring  Training  Materials  (E-I3A) 


J.2.2  Requirements  for  DCR.  The  DCR  should  have  the  ability  to 
administer  surveys  individually  or  in  a  group  administration. 


J-2.3  Frequency  of  Data  Collection.  Data  collected  via  Accept¬ 
ance  Surveys  will  be  collected  once  during  SLT&E  from  the  train¬ 
ing  developers  and  twice  from  all  other  participants.  Data  from 
training  developers  will  be  collected  in  September  1988;  data 
from  other  participant  will  be  collected  in  January  and  May 
1989 — using  the  same  set  of  acceptance  survey  statements  both 
times.  The  purpose  of  repeating  the  acceptance  survey  admnistra- 
tion  is  to  capture  any  opinion  changes  that  might  occur  during 
the  interval . 
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Four  open  ended  questions  will  be  included  one  time  on  all 
acceptance  surveys.  The  four  items  to  be  included  on  the 
developers  one  time  administered  acceptance  survey  and  on  the 
second  administration  of  all  other  acceptance  surveys  are  as  fol¬ 
lows. 


WHAT  DO  YOU  BELIEVE  ARE  THE  STRENGTHS  OF  AOTS? 

WHAT  DO  YOU  BELIEVE  ARE  THE  WEAKNESSES  OF  AOTS? 

HOW  WOULD  YOU  IMPROVE  AOTS? 

WHAT  CAPABILITIES  WOULD  YOU  LIKE  ADDED  TO  AOTS? 

J.2.4  Letters  of  Request.  Prior  to  administering  the  surveys, 
the  DCR  will  submit  a  letter  to  the  Squadron  Commander  requesting 
all  AOTS  participants  attend  the  Survey  administration  event. 
Squadron  Commanders  and  locations  are  found  in  Attachment  1  to 
Appendix  J. 

A  Memo  for  Record  will  be  sent  to  Major  Blackhurst,  advising  him 
that  the  1ST  will  be  responding  to  the  Developers  Survey.  The 
following  letter  may  be  used  as  a  model  for  the  letter  to 
Squadron  Commanders: 
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SAMPLE  LETTER  FOR  SCHEDULING  PERSONNEL  FOR  SLT&E  SURVEYS 


REPLY  TO 

ATTN  OF:  AFHRL/OL-AK 

SUBJECT:  AOTS  Survey  Administration 


TO:  67  AGS/CC 


1.  Request  the  following  personnel  be  scheduled  to  report  to  the 
Base  Theater  at  1300  hrs  on  29  Feb  1988.  The  individuals  will 
be  asked  to  respond  to  surveys  regarding  their  involvement  with 
the  prototype  Advanced  OJT  System  (AOTS) .  These  appointments 
should  take  approximately  1  hour. 


Captain  Hays,  Jeffrey  D. 
CMSgt  Singleton,  Joseph 
SMSgt  Dreher,  Steven  L. 
SMSgt  Gosc,  Robert  L. 
SMSgt  D'Amico,  James 
MSgt  Elliott,  Thomas  D. 
TSgt  Brooks,  David  W. 
SSgt  Hand,  Betsy 
A1C  Leger,  Lynn 
Amn  West,  James  J. 

AB  Sturdevant,  Wayne  A. 


FR000923847 
FR555223671 
FR843920199 
FR321664335 
FR631226734 
FR773400812 
FR543889201 
FR211211211 
FR23 19878 85 
FR012994999 
FR460403233 


12th  AMU, 
12th  AMU, 
12th  AMU, 
12th  AMU, 
12th  AMU, 
12th  AMU, 
12th  AMU, 
12th  AMU, 
12th  AMU, 
12th  AMU, 
12th  AMU, 


B  Flight 
A  Flight 
B  Flight 
A  Flight 
B  Flight 
B  Flight 
A  Flight 
A  Flight 
B  Flight 
A  Flight 
B  Flight 


JACK  L.  BLACKHURST,  Major,  USAF 
Commander,  AFHRL/OL-AK 
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The  DCR  will  administer 


J.2.5  Administration  of  Surveys. 

the  surveys.  The  DCR  will: 

-  Obtain  a  sufficient  number  of  #  2  pencils,  OMR  sheets 
and  Acceptance  Surveys  (Supervisor,  Training  Manager, 
Trainee,  and  Trainer/Evaluator) ,  for  distribution  to  the 
scheduled  AOTS  participants.  (The  Commander  Surveys  will 
PROBABLY  be  administered  separately) 

-  Attend  and  supervise  the  administration  event. 

—  Administer  the  surveys  in  a  room  free  from  distractions 
—  Distribute  the  surveys  and  OMR  sheets  to  each 
participant 

—  Provide  instructions  for  completing  the  surveys 
(see  sample  instructions  below) 

—  Allow  the  respondents  to  leave  and  return  to  the 
administration  area  without  the  surveys  (Surveys 
should  remain  with  the  DCR  while  respondents  are  absent 
from  the  survey  administration  area) 

—  Answer  any  questions  pertaining  to  the  completion  of 
the  forms 

—  Review  ALL  OMR  sheets  before  the  respondents  leave 
the  administration  area  and  ensure  that: 

The  correct  SSAN  was  entered 

The  correct  administration  time  was  entered  (1  for 
Jan,  2  for  May) 

The  correct  survey  number  was  entered 
All  question  and  statement  responses  were  entered 
—  Collect  all  pencils,  surveys  and  OMR  sheets  after 
the  surveys  have  been  completed 

SUGGESTIONS  FOR  ADMINISTERING  SURVEYS 

-  Allow  for  errors  (bring  extra  copies  of  surveys  and  OMR 
sheets ) 

-  Administer  the  surveys  in  a  classroom,  conference  room  or 
theatre 

-  Apply  a  ratio  of  one  DCR  to  approximately  15  respondents 

-  Administer  surveys  to  personnel  from  multiple  workcenters 
at  a  central  location  when  possible 

-  Read  the  following  SAMPLE  INSTRUCTIONS  to  the  respondents 

THE  PURPOSE  OF  THIS  SURVEY  IS  TO  DETERMINE  HOW  WELL  PERSON¬ 
NEL  IN  THE  WORKCENTERS,  WHO  HAVE  OPERATED  THE  AOTS,  ACCEPT  THE 
SYSTEM.  USE  THE  ACCOMPANYING  OMR  SHEET  AND  INDEPENDENTLY  RESPOND 
TO  ALL  QUESTIONS  AND  STATEMENTS.  NOTE  THAT  SOME  STATEMENTS  MAY 
APPEAR  TO  BE  IDENTICAL  TO  OTHERS  BUT  AFTER  CLOSER  EXAMINATION  YOU 
WILL  SEE  THAT  EACH  STATEMENT  REFERS  TO  A  DIFFERENT  TOPIC.  READ 
EACH  STATEMENT  CAREFULLY  BECAUSE  SOME  STATEMENTS  FAVOR  AOTS  AND 
SOME  FAVOR  CONVENTIONAL  OJT.  AFTER  YOU  HAVE  RESPONDED  TO  ALL  OF 
THE  STRUCTURED  STATEMENTS,  USE  THE  BACK  OF  THE  SURVEY  TO  REPORT 
ANY  COMMENTS  OR  SUGGESTIONS  THAT  YOU  HAVE  ABOUT  AOTS.  BE  HONEST 
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AND  TAKE  AS  LONG  AS  YOU  WANT  TO  RESPOND  TO  THE  SURVEY. 


J.2.6  Time  Required  for  Data  Collection.  Approximate  times  for 
each  administration  of  the  surveys  are  as  follows. 

1.5  hr.  per  workcenter — AFHRL  distribute/collect  surveys 

1.0  hr.  per  workcenter — Input  data  into  AOTS  data  table 

1.0  hr.  per  individual — Complete  surveys 

J.2.7  Disposition  of  Data.  After  the  surveys  have  been  col¬ 
lected,  DCR  will  conduct  a  quick-look  analysis  of  the  OMR  sheets 
to  insure  the  integrity  of  the  data  entered  by  the  respondents. 
The  following  plan  will  be  followed: 

-  Verify  that  all  SSANs  are  entered  and  that  the  SSANs 
correspond  with  the  names  and  SSANs  on  the  list  of 
incumbents. 

-  Verify  that  all  response  marks  on  each  OMR  sheet  are 
within  the  tolerance  limits  for  the  optical  scanner. 

-  Verify  that  each  form's  demographic  data  meets  the  stand¬ 
ards  set  on  the  sample  OMR  sheet  for  the  same  sur¬ 
vey. 

If  records  are  found  to  be  out  of  tolerance  or  with  error, 
the  DCR  will  return  the  OMR  sheets  to  the  respondent  for 
immediate  correction. 

Accurate  records  will  be  given  to  the  Data  Input 
Representative  (DIR)  for  data  input. 
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SUPERVISOR  ACCEPTANCE  SURVEY 


DIRECTIONS :  On  the  accompanying  answer  sheet,  please  enter; 
your  NAME  in  the  “NAME  GRID'*  block 

your  SSAN  in  columns  1-9  of  the  "NUMERIC  GRID"  block 
number  08  in  columns  24/25  of  the  "NUMERIC  GRID"  block 

Using  the  following  set  of  response  options  indicate  your 
response  to  each  statement  by  filling  in  the  oval,  on  the  answer 
sheet,  that  corresponds  to  the  letter  that  represents  your 
opinion. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

1*  HAVING  THE  TASK  ELEMENTS  (SUBTASKS,  ACTIVITIES,  SUPPORTING 
SKILLS  AND  KNOWLEDGE,  SUPPORT  EQUIPMENT,  ETC.)  LISTED  ON  THE  AOTS 
MASTER  TASK  LIST  HELPS  ME  MAKE  DECISIONS  WHEN  PLANNING  TRAINING 
AND  EVALUATION  EVENTS. 

2.  HAVING  THE  TASK  ELEMENTS  (SUBTASKS,  ACTIVITIES,  SUPPORTING 
SKILLS  AND  KNOWLEDGE,  SUPPORT  EQUIPMENT,  ETC.)  LISTED  ON  THE  AOTS 
MASTER  TASK  LIST  HELPS  ME  WHEN  I  AM  CONDUCTING  TRAINING. 

3.  OTHER  TRAINING  REQUIREMENTS  (FOR  EXAMPLE,  CONTINGENCY  TASK 
TRAINING,  ANCILLARY  TRAINING,  ADDITIONAL  DUTY  TRAINING,  AND 
CAREER  DEVELOPMENT  COURSES)  ARE  EASILY  IDENTIFIED  USING  THE  AOTS. 

4.  THE  AOTS  MAKES  IT  EASY  FOR  ME  TO  IDENTIFY  AIRMEN  TRAINING 
AND  EVALUATION  REQUIREMENTS. 

5.  THE  SEQUENCE  FOR  TRAINING,  DELIVERED  VIA  THE  AOTS,  CAN  BE 
EASILY  CHANGED  TO  ACCOMMODATE  UNSCHEDULED  TRAINING  OPPORTUNITIES 
OR  HIGHER  PRIORITIES  FOR  TRAINING. 

6.  USING  GENERIC  POSITION  TASK  REQUIREMENTS  (GPTR)  LISTS  AS  A 
BASELINE  TO  DEVELOP  OPERATIONAL  POSITION  TASK  REQUIREMENTS  (OPTR) 
LISTS  COMPLICATES  THE  PROCESSS  OF  IDENTIFYING  PERPriRMawn? 
REQUIREMENTS  FOR  AIRMEN. 

7.  TASKS  ON  OPERATIONAL  POSITION  TASK  REQUIREMENTS  (OPTR)  LISTS 
ARE  EASILY  RANK-ORDERED  WHEN  IT  IS  NECESSARY  TO  OVERRIDE  THE 
AUTOMATIC  RANK-ORDERING  FUNCTION. 
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SUPERVISOR  ACCEPTANCE  SURVEY 

Use  the  following  set  of  response  options  indicate  your  response 
to  each  statement. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

8.  TRAINING  REQUIREMENTS  ON  THE  INDIVIDUAL  TRAINING  REQUIREMENTS 
(ITR)  LISTS  ARE  EASILY  RANK-ORDERED  WHEN  IT  IS  NECESSARY  TO  OVER¬ 
RIDE  THE  AUTOMATIC  RANK-ORDERING  FUNCTION. 

9.  TASKS  CONTAINED  ON  THE  AOTS  MASTER  TASK  LIST  ARE  WELL 
DEFINED. 

10.  THE  MASTER  TASK  LIST  CONTAINS  A  COMPLETE  LISTING  OF  ALL  TASKS 
PERFORMED  IN  MY  WORK  CENTER. 

11.  THE  MASTER  TASK  LIST  EDITOR  IS  EASY  TO  USE. 

12.  THE  AOTS  PUBLICATION  REFERENCE  DATA  PINPOINT  EXACTLY  WHERE  A 
PERSON  CAN  FIND  INFORMATION  RELATED  TO  A  PARTICULAR  TASK. 

13.  WHEN  TASKS  ON  GPTR  LISTS  ARE  RANK-ORDERED  AUTOMATICALLY, 
THE  TASKS  ARE  ARRANGED  IN  THE  APPROXIMATE  ORDER  IN  WHICH  THE 
TASKS  WOULD  NORMALLY  BE  TRAINED  IN  THE  WORKCENTER. 

14.  WHEN  TASKS  ON  OPTR  LISTS  ARE  RANK-ORDERED  AUTOMATICALLY,  THE 
TASKS  ARE  ARRANGED  IN  THE  APPROXIMATE  ORDER  IN  WHICH  THE  TASKS 
WOULD  NORMALLY  BE  TRAINED. 

15.  THE  AOTS  AIRMAN  TRAINING  RECORD  CONTAINS  ALL  DATA  REQUIRED 
TO  DETERMINE  THE  POSITION  QUALIFICATION  REQUIREMENTS  FOR  AN  AIR¬ 
MAN. 


16.  THE  AOTS  MAKES  IT  EASY  TO  COMPARE  AIRMEN  TRAINING  COMPLE¬ 
TIONS  TO  POSITION  TRAINING  REQUIREMENTS. 

17.  THE  AOTS  QUALIFICATION  ASSESSMENT  PROCESS  PROVIDES  AN 
INACCURATE  LIST  OF  INDIVIDUAL  TRAINING  REQUIREMENTS  (ITRs) . 

18.  WHEN  TRAINING  REQUIREMENTS  LISTED  ON  AN  ITR  ARE  RANK-ORDERED 
AUTOMATICALLY,  THE  REQUIREMENTS  ARE  ARRANGED  IN  THE  APPROXIMATE 
ORDER  IN  WHICH  THE  REQUIREMENTS  WOULD  NORMALLY  BE  TRAINED. 
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SUPERVISOR  ACCEPTANCE  SURVEY 


Use  the  following  set  of  response  options  indicate  your  response 
to  each  statement. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

19.  THE  AOTS  ACCRUATELY  IDENTIFIES  COMMON  TRAINING  REQUIREMENTS 
FOR  GROUPS  OF  AIRMEN. 

20.  THE  AOTS  ACCURATELY  IDENTIFIES  MULTIPLE  TRAINING  REQUIRE¬ 
MENTS  FOR  A  SINGLE  AIRMAN. 

21.  WITH  AOTS,  IT  IS  EASY  TO  OVERRIDE  THE  AUTOMATED  TRAINING  AND 
EVALUATION  ASSIGNMENT  FUNCTION. 

22.  THE  AOTS  AIRMAN  TRAINING  RECORD  (ATR)  EDITOR  ACCURATELY 
TRACKS  AN  AIRMAN'S  TRAINING  PROGRESS. 

23.  THE  AOTS  GENERATES  EASILY  UNDERSTOOD  NOTICES  TO  INDICATE 
WHEN  AIRMEN  HAVE  BEEN  SCHEDULED  TO  ATTEND  TRAINING  CONDUCTED  OUT¬ 
SIDE  OF  THE  WORKCENTER. 

24.  TRAINEE  PROGRESS  REPORTS  GENERATED  BY  AOTS  PROVIDE  ALL  DATA 
NECESSARY  TO  DETERMINE  TRAINING  STATUS. 

25.  THE  AOTS  GENERATES  ACCURATE  SCHEDULES  FOR  PENDING  TRAINING 
AND  EVALUATION  EVENTS. 

26.  THE  "USER"  TRAINING  PROVIDED  PRIOR  TO  THE  AOTS  BEING  IMPLE¬ 
MENTED  WITHIN  MY  WORKCENTER  ADEQUATELY  PREPARED  ME  TO  OPERATE  THE 
AOTS  MANAGEMENT  FUNCTIONS. 

27.  THE  "USER"  TRAINING  PROVIDED  PRIOR  TO  THE  IMPLEMENTATION  OF 
AOTS  ADEQUATELY  PREPARED  ME  TO  MAINTAIN  MANAGEMENT  DATA. 

28.  THE  "USER"  TRAINING  PROVIDED  PRIOR  TO  THE  IMPLEMENTATION  OF 
AOTS  ADEQUATELY  PREPARED  ME  TO  OBTAIN  INFORMATION  FROM  THE 
MANAGEMENT  DATA  FILES. 

29.  THE  AOTS  IS  DIFFICULT  TO  OPERATE  WHEN  REVIEWING  COMPUTER- 
DELIVERED  TRAINING  MATERIALS  (i.e.  CAI)  ON-LINE. 

30.  THE  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  ARE  DISPLAYED 
IN  A  LOGICAL,  EASY-TO-UNDERSTAND  ORDER. 
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SUPERVISOR  ACCEPTANCE  SURVEY 

Use  the  following  set  of  response  options  indicate  your  response 
to  each  statement. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

31.  THE  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  CONTAIN  INAC¬ 
CURATE  INFORMATION. 

32.  THE  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  CONTAIN  CUR¬ 
RENT  INFORMATION. 

33.  THE  COMPUTER  EQUIPMENT  AVAILABLE  TO  MY  WORKCENTER  FOR  THE 
DELIVERY  OF  TRAINING  IS  SUFFICIENT  TO  SUPPORT  THE  NEEDS  OF  THE 
PERSONNEL  ASSIGNED  TO  MY  WORKCENTER. 

34.  THE  CONTROLS  ESTABLISHED  WITHIN  THE  AOTS  PREVENT  UN¬ 
AUTHORIZED  PERSONS  FROM  GAINING  ACCESS  TO  EVALUATION  MATERIALS. 

35.  THE  USE  OF  THE  AOTS  EVAIUATION  PROCEDURES  IN  MY  WORKCENTER 
HELPS  TO  ENSURE  THAT  AIRMEN  ARE  POSITION  QUALIFIED. 

36.  ADMINISTERING  PRE-TRAINING  EVALUATIONS  ON  THE  TASKS  THE 
TRAINEES  MUST  PERFORM  HELPS  TO  IDENTIFY  TRAINING  NEEDS. 

37.  THE  USE  OF  AOTS  EVALUATIONS  INCREASES  THE  PRODUCTIVITY  OF 
PERSONNEL  ASSIGNED  TO  MY  WORKCENTER. 

38.  EVALUATION  MATERIALS  REQUIRED  FOR  OFF-LINE  USE  ARE  EASILY 
OBTAINED  FROM  THE  AOTS  FOR  AUTHORIZED  PERSONNEL. 

39.  THE  USE  OF  AOTS  PROCEDURES  FOR  EVALUATING  PERSONNEL  WITHIN 
WORKCENTERS  HELPS  TO  DETERMINE  TRAINING  PROGRAM  EFFECTIVENESS. 

40.  THE  AOTS  PROCEDURES  FOR  SCORING  TESTS  THAT  ARE  ADMINISTERED 
ON-LINE  PROVIDE  IMMEDIATE  FEEDBACK  (TEST  SCORE)  TO  THE  TRAINEE. 

41.  THE  "USER"  TRAINING  PROVIDED  PRIOR  TO  THE  AOTS  BEING  IMPLE¬ 
MENTED  WITHIN  WORKCENTERS  DID  NOT  ADEQUATELY  PREPARE  PERSONNEL  TO 
OPERATE  THE  TESTING  FUNCTIONS. 


Figure  J.2-1 
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TRAINING  MANAGER  ACCEPTANCE  SURVEY 


DIRECTIONS :  On  the  accompanying  answer  sheet,  please  enter; 
your  NAME  in  the  “NAME  GRID"  block 

your  SSAN  in  columns  1-9  of  the  "NUMERIC  GRID"  block 
number  10  in  columns  24/25  of  the  "NUMERIC  FRID"  block 

Using  the  following  set  of  response  options,  indicate  your 
response  to  each  statement  by  filling  in  the  oval,  on  the  answer 
sheet,  that  corresponds  to  the  letter  that  represents  your 
opinion. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

1.  HAVING  THE  TASK  ELEMENTS  (SUBTASKS,  ACTIVITIES,  SUPPORTING 
SKILLS  AND  KNOWLEDGE,  SUPPORT  EQUIPMENT,  ETC.)  LISTED  ON  THE  AOTS 
MASTER  TASK  LIST  HELPS  ME  TO  MAKE  DECISIONS  WHEN  PLANNING  TRAIN¬ 
ING  AND  EVALUATION  EVENTS. 

2.  HAVING  THE  TASK  ELEMENTS  (SUBTASKS,  ACTIVITIES,  SUPPORTING 
SKILLS  AND  KNOWLEDGE,  SUPPORT  EQUIPMENT,  ETC.)  LISTED  ON  THE  AOTS 
MASTER  TASK  LIST  HELPS  ME  WHEN  I  AM  CONDUCTING  TRAINING. 

3.  OTHER  TRAINING  REQUIREMENTS  (FOR  EXAMPLE,  CONTINGENCY  TASK 
TRAINING,  ANCILLARY  TRAINING,  ADDITIONAL  DUTY  TRAINING,  AND 
CAREER  DEVELOPMENT  COURSES)  ARE  EASILY  IDENTIFIED  USING  THE  AOTS. 

4.  THE  AOTS  MAKES  IT  EASY  FOR  ME  TO  IDENTIFY  AIRMEN  TRAINING  AND 
EVALUATION  REQUIREMENTS. 

5.  THE  SEQUENCE  FOR  TRAINING,  DELIVERED  VIA  THE  AOTS,  CAN  BE 
EASILY  CHANGED  TO  ACCOMMODATE  UNSCHEDULED  TRAINING  OPPORTUNITIES 
OR  HIGHER  PRIORITIES. 

6.  USING  GENERIC  POSITION  TASK  REQUIREMENTS  (GPTR)  LISTS  AS  A 
BASELINE  TO  DEVELOP  OPERATIONAL  POSITION  TASK  REQUIREMENTS  (OPTR) 
LISTS  COMPLICATES  THE  PROCESS  OF  IDENTIFYING  PERFORMANCE  REQUIRE¬ 
MENTS  FOR  AIRMEN. 

7.  TASKS  ON  OPERATIONAL  POSITION  TASK  REQUIREMENTS  (OPTR)  LISTS 
ARE  EASILY  RANK-ORDERED  WHEN  IT  IS  NECESSARY  TO  OVERRIDE  THE 
AUTOMATIC  RANK-ORDERING  FUNCTION. 
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TRAINING  MANAGER  ACCEPTANCE  SURVEY 


Use  the  following  set  of  response  options,  indicate  your  response 
to  each  statement. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

8.  TRAINING  REQUIREMENTS  ON  THE  INDIVIDUAL  TRAINING  REQUIRE¬ 
MENTS  (ITR)  LISTS  ARE  EASILY  RANK-ORDERED  WHEN  IT  IS  NECESSARY  TO 
OVERRIDE  THE  AUTOMATIC  RANK-ORDERING  FUNCTION. 

9 .  TASKS  CONTAINED  ON  THE  AOTS  MASTER  TASK  LIST  ARE  WELL 
DEFINED. 

10.  THE  AOTS  PUBLICATION  REFERENCE  DATA  PINPOINT  EXACTLY  WHERE  A 
PERSON  CAN  FIND  INFORMATION  RELATED  TO  A  PARTICULAR  TASK. 

11.  WHEN  TASKS  ON  GPTR  LISTS  ARE  RANK-ORDERED  AUTOMATICALLY, 
THE  TASKS  ARE  ARRANGED  IN  THE  APPROXIMATE  ORDER  IN  WHICH  THE 
TASKS  WOULD  NORMALLY  BE  TRAINED. 

12.  WHEN  TASKS  ON  OPTR  LISTS  ARE  RANK-ORDERED  AUTOMATICALLY, 
THE  TASKS  ARE  ARRANGED  IN  THE  APPROXIMATE  ORDER  IN  WHICH  THE 
TASKS  WOULD  NORMALLY  BE  TRAINED. 

13.  THE  AOTS  AIRMAN  TRAINING  RECORD  CONTAINS  ALL  DATA  REQUIRED 
TO  DETERMINE  THE  POSITION  QUALIFICATION  REQUIREMENTS  FOR  AN  AIR¬ 
MAN. 


14.  THE  AOTS  AIRMAN  TRAINING  RECORDS  MAKE  IT  EASY  TO  COMPARE 
AIRMEN  TRAINING  COMPLETIONS  TO  POSITION  TRAINING  REQUIREMENTS. 

15.  THE  AOTS  QUALIFICATION  ASSESSMENT  PROCESS  PROVIDES  AN 
INACCURATE  LIST  OF  INDIVIDUAL  TRAINING  REQUIREMENTS. 

16.  WHEN  TRAINING  REQUIREMENTS  LISTED  ON  AN  ITR  ARE  RANK-ORDERED 
AUTOMATICALLY,  THE  REQUIREMENTS  ARE  ARRANGED  IN  THE  APPROXIMATE 
ORDER  IN  WHICH  THE  REQUIREMENTS  WOULD  NORMALLY  BE  TRAINED. 

17.  THE  AOTS  ACCURATELY  IDENTIFIES  COMMON  TRAINING  REQUIREMENTS 
FOR  GROUPS  OF  AIRMEN. 

18.  THE  AOTS  ACCURATELY  IDENTIFIES  MULTIPLE  TRAINING  REQUIRE¬ 
MENTS  NEEDED  FOR  A  SINGLE  AIRMAN. 
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TRAINING  MANAGER  ACCEPTANCE  SURVEY 


Use  the  following  set  of  response  options,  indicate  your  response 
to  each  statement. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

19.  WITH  AOTS ,  IT  IS  EASY  TO  OVERRIDE  THE  AUTOMATED  TRAINING  AND 
EVALUATION  ASSIGNMENT  FUNCTION. 

20.  THE  AOTS  AIRMAN  TRAINING  RECORD  (ATR)  EDITOR  ACCURATELY 
TRACKS  AN  AIRMAN'S  TRAINING  PROGRESS. 

21.  THE  AOTS  GENERATES  EASILY  UNDERSTOOD  NOTICES  TO  INDICATE 
WHEN  AIRMEN  HAVE  BEEN  SCHEDULED  TO  ATTEND  TRAINING  CONDUCTED  OUT¬ 
SIDE  OF  THE  WORKCENTER. 

22.  TRAINEE  PROGRESS  REPORTS  GENERATED  BY  AOTS  PROVIDE  ALL  DATA 
NECESSARY  TO  DETERMINE  TRAINING  STATUS. 

23.  THE  AOTS  GENERATES  ACCURATE  SCHEDULES  FOR  PENDING  TRAINING 
AND  EVALUATION  EVENTS. 

24.  THE  "USER"  TRAINING  PROVIDED  PRIOR  TO  THE  AOTS  BEING  IMPLE¬ 
MENTED  WITHIN  WORKCENTERS  ADEQUATELY  PREPARED  PERSONNEL  TO 
OPERATE  THE  AOTS  MANAGEMENT  FUNCTIONS. 

25.  THE  "USER"  TRAINING  PROVIDED  PRIOR  TO  THE  IMPLEMENTATION  OF 
AOTS  PREPARED  PERSONNEL  TO  MAINTAIN  MANAGEMENT  DATA. 

26.  THE  "USER"  TRAINING  PROVIDED  PRIOR  TO  THE  IMPLEMENTATION  OF 
AOTS  PREPARED  PERSONNEL  TO  OBTAIN  INFORMATION  FROM  THE  MANAGEMENT 
DATA  FILES. 

27.  THE  CONTROLS  ESTABLISHED  WITHIN  THE  AOTS  PREVENT  UN¬ 
AUTHORIZED  PERSONS  FROM  GAINING  ACCESS  TO  EVALUATION  MATERIALS. 

28.  THE  USE  OF  THE  AOTS  EVALUATION  PROCEDURES  IN  THE  WORKCENTERS 
HELPS  TO  ENSURE  THAT  AIRMEN  ARE  POSITION  QUALIFIED. 

29.  THE  USE  OF  AOTS  EVALUATION  PROCEDURES  INCREASES  THE  PRODUC¬ 
TIVITY  OF  PERSONNEL  ASSIGNED  TO  WORKCENTERS: 

30.  EVALUATION  MATERIALS  REQUIRED  FOR  OFF-LINE  USE  ARE  EASILY 
OBTAINED  FROM  THE  AOTS  FOR  AUTHORIZED  PERSONNEL. 
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TRAINING  MANAGER  ACCEPTANCE  SURVEY 


Use  the  following  set  of  response  options,  indicate  your  response 
to  each  statement. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

31.  THE  USE  OF  AOTS  PROCEDURES  FOR  EVALUATING  PERSONNEL  WITHIN 
WORKCENTERS  HELPS  TO  DETERMINE  TRAINING  PROGRAM  EFFECTIVENESS. 

32.  THE  AOTS  PROCEDURES  FOR  SCORING  TESTS  THAT  ARE  ADMINISTERED 
ON-LINE  PROVIDES  IMMEDIATE  FEEDBACK  (TEST  SCORE)  TO  THE  TRAINEE. 

33.  THE  "USER"  TRAINING  PROVIDED  PRIOR  TO  THE  AOTS  BEING  IMPLE¬ 
MENTED  WITHIN  WORKCENTERS  DID  NOT  ADEQUATELY  PREPARE  PERSONNEL  TO 
OPERATE  THE  TESTING  FUNCTIONS. 


Figure  J.2-2 
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TRAINER/EVALUATOR  ACCEPTANCE  SURVEY 


DIRECTIONS :  On  the  accompanying  answer  sheet,  please  enter; 
your  NAME  in  the  "NAME  GRID  block 

your  SSAN  in  columns  1-9  of  the  ‘'NUMERIC  GRID1'  block 
number  09  in  columns  24/25  of  the  "NUMERIC  GRID"  block 

Using  the  following  set  of  response  options,  indicate  your 
response  to  each  statement  by  filling  in  the  oval,  on  the  answer 
sheet,  that  corresponds  to  the  letter  that  represents  your  op- 
nion. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

1.  THE  AOTS  IS  DIFFICULT  TO  OPERATE  WHEN  REVIEWING  COMPUTER- 
DELIVERED  TRAINING  MATERIALS  (i.e.  CAI)  ON-LINE, 

2.  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  ARE  DISPLAYED  IN  A 
LOGICAL,  EASY-TO-UNDERSTAND  ORDER. 

3.  THE  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  CONTAIN  INAC¬ 
CURATE  INFORMATION. 


4.  THE  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  CONTAIN  CURRENT 
INFORMATION . 


5.  THE  COMPUTER  EQUIPMENT  AVAILABLE  TO  MY  WORKCENTER  FOR  THE 
DELIVERY  OF  TRAINING  IS  SUFFICIENT  TO  SUPPORT  THE  NEEDS  OF  THE 
PERSONNEL  ASSIGNED  TO  MY  WORKCENTER. 


6.  EVALUATION  MATERIALS  REQUIRED  FOR  OFF-LINE  USE  ARE  EASILY  OB¬ 
TAINED  FROM  THE  AOTS  FOR  AUTHORIZED  PERSONNEL. 


7.  THE  USE  OF  AOTS  PROCEDURES  FOR  EVALUATING  PERSONNEL  WITHIN  MY 
WORKCENTER  HELPS  TO  DETERMINE  TRAINING  PROGRAM  EFFECTIVENESS. 


R  .  rPT4T?  & OTG  ‘DDAPC,r\TTDT?0  PAT)  CAAnTim  rr.ppmn  mm  m  unn  ■» 
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ON-LINE  PROVIDE  IMMEDIATE  FEEDBACK  (TEST  SCORES)  TO  THE  TRAINEE. 


9.  THE  USER  TRAINING  PROVIDED  PRIOR  TO  THE  AOTS  BEING  IMPLE¬ 
MENTED  WITHIN  MY  WORKCENTER  DID  NOT  ADEQUATELY  PREPARE  PERSONNEL 
TO  OPERATE  THE  TESTING  FUNCTIONS. 


Figure  J.2-3 
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TRAINEE  ACCEPTANCE  SURVEY 


DIRECTIONS:  On  the  accompanying  answer  sheet,  please  enter; 
your  NAME  in  the  "NAME  GRID"  block 

your  SSAN  in  columns  1-9  of  the  "NUMERIC  GRID"  block 
number  11  in  columns  24/25  of  the  "NUMERIC  GRID"  block 

Using  the  following  set  of  response  options,  indicate  your 
response  to  each  statement  by  filling  in  the  oval,  on  the  answer 
sheet,  that  corresponds  to  the  letter  that  represents  your 
opinion. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

1.  THE  AOTS  IS  DIFFICULT  TO  OPERATE  WHEN  REVIEWING  COMPUTER 
DELIVERED  TRAINING  MATERIALS  (i.e.  CAI)  ON-LINE. 

2.  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  ARE  DISPLAYED  IN  A 
LOGICAL,  EASY-TO-UNDERSTAND  ORDER. 

3.  THE  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  CONTAIN  INAC¬ 
CURATE  INFORMATION. 

4.  THE  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  CONTAIN  CURRENT 
INFORMATION. 

5.  THE  USE  OF  TRAINING  MATERIALS  DELIVERED  VIA  THE  AOTS  HAS  IN¬ 
CREASED  MY  ABILITY  TO  PERFORM  RELATED  TASKS. 

6.  THE  COMPUTER  EQUIPMENT  AVAILABLE  TO  MY  WORKCENTER  FOR  THE 
DELIVERY  OF  TRAINING  IS  SUFFICIENT  TO  SUPPORT  THE  NEEDS  OF  THE 
PERSONNEL  ASSIGNED  TO  MY  WORKCENTER. 

7.  THE  "USER”  TRAINING  PROVIDED  PRIOR  TO  THE  AOTS  BEING  IMPLE¬ 
MENTED  WITHIN  MY  WORKCENTER  DID  NOT  ADEQUATELY  PREPARE  ME  TO 
OPERATE  THE  ON-LINE  TESTING  FUNCTIONS. 

8.  THE  "USER"  TRAINING  PROVIDED  PRIOR  TO  THE  AOTS  BEING  IMPLE¬ 
MENTED  WITHIN  MY  WORKCENTER  ADEQUATELY  PREPARED  ME  TO  OPERATE  THE 
TRAINING  DELIVERY  FUNCTIONS. 

9.  I  WAS  GIVEN  ADEQUATE  INSTRUCTIONS  ON  THE  OPERATION  OF  AOTS  TO 
BE  ABLE  TO  PROCEED  SMOOTHLY  THROUGH  THE  COMPUTER  ASSISTED  IN¬ 
STRUCTION  (CAI)  LESSON (S) . 
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TRAINEE  ACCEPTANCE  SURVEY 

Use  the  following  set  of  response  options,  indicate  your  response 
to  each  statement. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

10.  I  WAS  GIVEN  ADEQUATE  INSTRUCTIONS  TO  UNDERSTAND  AND  INTERACT 
WITH  THE  COMPUTER  PROMPTS. 

11.  I  WAS  GIVEN  ADEQUATE  INSTRUCTION  TO  BE  ABLE  TO  COMPLETE  THE 
CAI  KNOWLEDGE  TEST  QUESTIONS  PRESENTED  ON  THE  SCREEN. 


Figure  J.2-4 
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COMMANDER  ACCEPTANCE  SURVEY 


DIRECTIONS :  On  the  accompanying  answer  sheet,  please  enter; 
your  NAME  in  the  "Name  Grid5'  block 

your  SSAN  in  columns  1-9  of  the  ‘'NUMERIC  GRID"  block 
number  07  in  columns  24/25  of  the  "NUMERIC  GRID"  block. 

Using  the  following  set  of  response  options,  indicate  your 
response  to  each  statement  by  filling  in  the  oval,  on  the  answer 
sheet,  that  corresponds  to  the  letter  that  represents  your 
opinion. 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

DISAGREE 

NEUTRAL 

STRONGLY 

AGREE 

1.  THE  aots  generated  reports  provide  all  the  data  required  to 

DETERMINE  THE  CURRENT  TRAINING  STATUS  OF  INDIVIDUAL  AIRMEN. 

2.  THE  AOTS  GENERATED  REPORTS  PROVIDE  ALL  THE  DATA  REQUIRED  TO 
DETERMINE  THE  CURRENT  TRAINING  STATUS  OF  INDIVIDUAL  WORKCENTERS . 

3.  THE  AOTS  GENERATED  REPORTS  PROVIDE  ALL  THE  DATA  REQUIRED  TO 
DETERMINE  TRAINING  EFFECTIVENESS . 

4.  THE  AOTS  GENERATED  REPORTS  PROVIDE  ALL  THE  DATA  REQUIRED  TO 
DETERMINE  TRAINING  EFFICIENCY. 

5.  THE  AOTS  GENERATED  REPORTS  ARE  PRESENTED  IN  AN  ACCEPTABLE 
FORMAT. 

6.  THE  DATA  BEING  COLLECTED  ARE  SUFFICIENT  TO  DETERMINE  WHETHER 
AOTS  IS  IMPROVING  THE  QUALITY  OF  TRAINING. 

7.  AIRMEN  ARE  MORE  CAPABLE  OF  MEETING  MISSION  REQUIREMENTS  WHEN 
TRAINED  UNDER  AOTS  THAN  WHEN  TRAINED  UNDER  CONVENTIONAL  OJT. 

Figure  J-2-5 
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BLANK  PAGE 


TO  BE  USED 

IF  ADDITIONAL  ITEMS  ARE  ADDED  TO  THE  COMMANDER  SURVEY 
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TRAINING  DEVELOPER  ACCEPTANCE  SURVEY 


DIRECTIONS:  On  the  accompanying  answer  sheet,  please  enter? 

your  NAME  in  the  "NAME  GRID"  block 

your  SSAN  in  columns  1-9  of  the  "NUMERIC  GRID"  block 
number  12  in  columns  24/25  of  the  "NUMERIC  GRID"  block 

Using  the  following  set  of  response  options/  indicate  your 
response  to  each  statement  by  filling  in  the  oval,  on  the  answer 
sheet,  that  corresponds  to  the  letter  that  represents  your 
opinion. 

A  B  C  D  E  F  G 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 


1.  I  DID  NOT  REQUIRE  A  COMPUTER  BACKGROUND  TO  UNDERSTAND  THE 
INFORMATION  DISPLAYED  ON  THE  AOTS  SCREENS. 

2.  AFTER  I  SELECTED  A  MENU  OPTION,  THE  AOTS  SYSTEM  DISPLAYED  THE 
SCREEN  THAT  I  EXPECTED  TO  SEE. 

3.  THE  AOTS  SCREEN  DISPLAYS  WERE  UNCLUTTERED. 

4.  THE  AOTS  SCREEN  DISPLAYS  GAVE  ME  ENOUGH  INFORMATION  (i.e., 
TASK  ID,  WORKCENTER  ID,  TEST  ITEM  ID)  TO  DO  MY  WORK  ACCURATELY. 

5.  THE  PROMPTS  SUCH  AS  (Q)uit,  (A)dd,  and  (E)dit  AT  THE  BOTTOM 
OF  THE  SCREEN  WERE  UNDERSTANDABLE. 

6.  AFTER  I  SELECTED  A  PROMPT  OPTION  SUCH  AS  (Q)uit,  (A)dd,  OR 
(E)dit,  THE  AOTS  SYSTEM  PERFORMED  THE  FUNCTION  THAT  I  EXPECTED  TO 
BE  PERFORMED. 

7.  OVERALL,  THE  ERROR  MESSAGES  WERE  DESCRIPTIVE  AND  HELPFUL. 

8.  THE  AOTS  EDITING  CAPABILITY  (BEING  ABLE  TO  CORRECT  OR  CHANGE 
ENTRIES)  WAS  DIFFICULT  TO  USE. 

9.  I  COULD  NOT  CONTROL  THE  AOTS  SYSTEM  AS  MUCH  AS  I  NEEDED. 
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TRAINING  DEVELOPER  ACCEPTANCE  SURVEY 


Use  the  following  set  of  response  options,  indicate  your  response 
to  each  statement. 

A  B  C  D  E  P  G 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 


10.  THE  AOTS  TASK  PUBLICATIONS  (TP)  EDITOR  ALLOWED  FOR  AN  EXACT 
LINKAGE  BETWEEN  THE  TASK  REFERENCE  AND  THE  TASK  ELEMENTS  (e.g., 
TERMINAL  TASK,  SUBTASK,  BEHAVIORAL  OBJECTIVE,  TEST  ITEM). 

11.  THE  AOTS  IDENTIFIED  TASK  AND  TEST  ELEMENTS  (e.g.  TERMINAL 
TASK,  SUBTASK,  BEHAVIORAL  OBJECTIVE,  AND  TEST  ITEMS)  THAT  MIGHT 
BE  AFFECTED  BY  A  PUBLICATION  CHANGE. 

12.  THE  AOTS  BEHAVIORAL  OBJECTIVE  EDITOR  WAS  EASY  TO  USE  IN  THE 
NOVICE  MODE. 

13.  THE  AOTS  BEHAVIORAL  OBJECTIVE  EDITOR  WAS  EASY  TO  USE  IN  THE 
EXPERT  MODE. 

14.  THE  AOTS  TEST  ITEM  BANK  EDITOR  WAS  EASY  TO  USE. 

15.  THE  AOTS  TEST  ITEM  BANK  (TIB)  EDITOR  EFFECTIVELY  STORED  THE 
TEST  ITEMS  (e.g.,  KNOWLEDGE  TEST  ITEMS,  PERFORMANCE  EVALUATION 
CHECK  LISTS,  AND  ORAL  TEST  GUIDES). 

16.  GRAPHICS  WERE  EASILY  ADDED  TO  THE  AOTS  TEST  ITEMS. 

17.  THE  AOTS  TEST  EDITOR  WAS  EASY  TO  USE. 

18.  THE  AOTS  TEST  EDITOR  PERMITTED  THE  CREATION  OF  PERFORMANCE 
TESTS . 


19.  THE  AOTS  TEST  EDITOR  PERMITTED  THE  CREATION  OF  KNOWLEDGE 
TESTS . 


20.  TEST  PARAMETERS  (i.e.,  ITEM  ANALYSIS,  ITEM  SCRAMBLING, 
INSTRUCTIONS)  WERE  EASILY  ESTABLISHED  USING  THE  AOTS  TEST  EDITOR. 


Figure  J.2-6 
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J.3  SLT&E  DEPLOYMENT  SURVEY 


J.3.1  Data  To  Be  Collected.  The  Deployment  Surveys  are 

designed  to  collect  opinions  from  AOTS  participants  on  whether 
the  AOTS  prototype  should  be  deployed  to  the  Air  Force  opera¬ 
tional  workcenters.  A  sample  survey  can  be  found  on  page  44 
(Figure  J.3-1).  SLT&E  Deployment  Surveys  consist  of  data  col¬ 
lected  from  four  sources  as  follows: 

Commanders 
Training  Managers 
Supervisors 
Trainers/Evaluators 

If  a  person  acts  as  both  a  Supervisor  and  an  Evaluator  or  as 
both  a  Supervisor  and  a  Trainer,  he/she  should  be  categorized  as 
a  Supervisor. 

Using  a  response  scale  ranging  from  A  to  G,  all  of  the 
respondents  are  asked  to  indicate  how  strongly  they  oppose  or 
favor  the  deployment  of  the  AOTS  prototype  to  Air  Force  opera¬ 
tional  work  centers  (E-25A) .  After  the  Deployment  Surveys  are 
administered,  structured  interview  responses  are  obtained  from 
all  commanders,  training  managers,  and  fifty  percent  of  the  su¬ 
pervisors,  trainers  and  evaluators.  The  supervisors,  trainers 
and  evaluators  are  randomly  identified  by  coding  every  other 
answer  sheet,  to  be  used  by  this  group  of  Deployment  Survey 
respondents,  BEFORE  the  administration  event  begins. 


J.3.2  Requirements  for  DCR.  The  DCR  must  have  the  ability  to 
administer  surveys  individually  or  in  a  group  administration  and 
be  able  to  conduct  a  one-on-one  interview.  The  DCRs  must  be  able 
to  ask  questions  and  simultaneously  record  accurate  interviewee 
responses.  Any  DCRs  who  do  not  possess  the  skills  required  to 
conduct  a  one-on-one  interview  session  must  be  trained  to  do  so. 

J-3.3  Frequency  of  Data  Collection.  The  deployment  surveys 
will  be  administered  one  time  towards  the  end  of  SLT&E  (May  1989) 

J-3.4  Letters  of  Request.  Prior  to  administering  the  surveys, 
the  DCR  will  submit  a  letter  to  the  Commander  requesting  all  AOTS 
participants  attend  the  Deployment  Survey  administration  event. 
Squadron  Commanders  and  locations  are  found  in  Attachment  1  to 
Appendix  J. 

J<3.5  Data  Collection.  The  DCR  will  administer  the  Deploy¬ 
ment  Survey  and  conduct  one-on-one  interviews.  The  interviews 
may  be  conducted  immediately  after  the  administration  of  the  sur¬ 
veys,  or  at  a  later  date  as  appropriate.  Interviewees  should  NOT 
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be  asked  to  wait  at  the  administration  site  longer  than  15 
minutes  after  the  completion  of  the  deployment  survey  administra¬ 
tion  before  being  interviewed.  Any  interviewee  who  can  not  be 
scheduled  to  be  interviewed  within  15  minutes  after  the  survey 
completion  should  be  scheduled  for  another  time. 

J.3.5.1  Survey  Administration.  The  DCR  will: 

-  Obtain  a  sufficient  number  of  #  2  pencils,  Interview 
Response  Sheets;  OMR  sheets,  and  Surveys  (Supervisor, 
Training  Manager,  and  Trainer/Evaluator)  for  distribution 
to  the  scheduled  AOTS  participants.  (Surveys  will  probably 
be  administered  separately  to  commanders) 

-  Attend  and  supervise  the  administration  event, 

—  Administer  the  surveys  in  a  room  free  from  distractions 
—  Distribute  the  surveys  and  OMR  sheets  to  each 
participant 

—  Provide  instructions  for  completing  the  surveys 
(see  sample  instructions  below) 

—  Allow  the  respondents  to  leave  and  return  to  the 
administration  area  without  the  surveys  (Surveys 
should  remain  with  the  DCR  while  respondents  are  absent 
from  the  survey  administration  area) 

—  Answer  any  questions  pertaining  to  the  completion  of 
the  survey  forms 

—  Review  ALL  OMR  sheets  before  the  respondents  leave 
the  administration  area  and  ensure  that: 

The  correct  SSAN  was  entered 
The  correct  Survey  number  was  entered 
All  question  and  statement  responses  were  entered 
—  Collect  all  pencils,  surveys  and  OMR  sheets  after 
the  surveys  have  been  completed 

SUGGESTIONS  FOR  ADMINISTERING  SURVEYS  AND  CONDUCTING  INTERVIEWS 

-  Allow  for  errors  (bring  extra  copies  of  surveys,  OMR 
sheets,  and  Interview  Response  Sheets) 

-  Administer  the  surveys  in  a  classroom,  conference  room  or 
theatre 

-  Apply  a  ratio  of  one  DCR  to  approximately  5  respondents 

-  Administer  surveys  to  personnel  from  multiple  workcenters 
at  a  central  location  if  possible 

-  Read  the  following  SAMPLE  INSTRUCTIONS  to  the  respondents 

THE  PURPOSE  OF  THIS  SURVEY  IS  TO  DETERMINE  WHETHER  PERSON¬ 
NEL  IN  THE  WORKCENTERS,  WHO  HAVE  OPERATED  THE  AOTS,  BELIEVE  THE 
AOTS  SHOULD  BE  DEPLOYED  TO  ALL  WORKCENTERS.  USING  THE  ACCOMPANY¬ 
ING  OMR  SHEET,  INDEPENDENTLY  RESPOND  TO  ALL  QUESTIONS  AND  STATE¬ 
MENTS.  NOTE  THAT  SOME  STATEMENTS  KAY  APPEAR  TO  BE  IDENTICAL  TO 
OTHERS  BUT  AFTER  CLOSER  EXAMINATION  YOU  WILL  SEE  THAT  EACH  STATE¬ 
MENT  REFERS  TO  A  DIFFERENT  TOPIC.  READ  EACH  STATEMENT 
CAREFULLY  BECAUSE  SOME  STATEMENTS  FAVOR  AOTS  AND  SOME  FAVOR  CON- 
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VENTIONAL  OJT.  AFTER  YOU  HAVE  RESPONDED  TO  ALL  OF  THE  STRUCTURED 
STATEMENTS,  USE  THE  BACK  OF  THE  SURVEY  TO  REPORT  ANY  COMMENTS  OR 
SUGGESTIONS  THAT  YOU  HAVE  ABOUT  AOTS.  BE  HONEST  AND  TAKE  AS  LONG 
AS  YOU  WANT  TO  RESPOND  TO  THE  SURVEY. 

J.3.5.2  Interview  Administration.  The  DCR  will  conduct  one-on- 
one  interviews  with  previously  selected  respondents.  The  DCR 
will  review  the  interviewee's  Deployment  Survey  OMR  sheet,  and 
using  a  copy  of  the  Deployment  Survey,  obtain  additional  informa¬ 
tion  for  any  items  that  are  NOT  answered  with  a  neutral  (D) 
response. 


If  a  response  falls  in  the  NEGATIVE  category,  the  interviewer 
will  substitute  the  words  between  the  parenthesis  with  the  ap¬ 
propriate  terminology  from  each  item  and  ask  the  follow 
questions: 

a.  WHY  DO  YOU  BELIEVE  THE  (OPTR  EDITOR  IS  NOT  CAPABLE  OF  BEING 
USED  IN  AN  OPERATIONAL  ENVIRONMENT)? 

b.  HOW  COULD  THE  (OPTR  EDITOR  BE  IMPROVED  TO  MAKE  IT  MORE 
FUNCTIONAL  IN  THE  WORKCENTERS) ? 


If  a  response  falls  in  the  POSITIVE  category,  the  interviewer 
will  substitute  the  words  between  the  parenthesis  with  the  ap¬ 
propriate  terminology  from  each  item  and  ask  the  follow 
questions: 

a.  WHY  DO  YOU  BELIEVE  THE  (OPTR  EDITOR  WILL  BE  A  SUCCESS  IN  THE 
OPERATIONAL  ENVIRONMENT) ? 

b.  WHAT  DO  YOU  BELIEVE  IS  THE  BEST  FEATURE  OF  THE  (OPTR 
EDITOR) ? 

If  more  than  three  lines  are  needed  for  a  response,  the  inter¬ 
viewers  will  use  the  back  side  of  the  respective  sheet  ensuring 
that  all  continued  response  statements  are  numbered  correctly. 

If  several  individuals  are  interviewed  simultaneously  in  the  same 
room,  the  DCRs  will  conduct  the  interviews  in  separate  parts  of 
the  room  to  provide  as  much  privacy  as  possible  to  the  inter¬ 
viewees. 

Each  DCR  will  interview  one  participant  at  a  time.  If  groups 
are  mixed,  Commanders  will  be  interviewed  first,  followed  by 
Training  Managers,  Supervisors  and  Trainers/Evaluators. 

The  DCRs  will  take  accurate  notes  of  participants'  comments 
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during  the  interview.  As  soon  as  possible  after  the  interview 
has  been  concluded  the  DCRs  will  expand  on  the  notes  to  ensure 
comprehensive  data  are  collected  and  recorded. 


J . 3 . 6  Time  Required  for  Deployment  Data  Collection . 


1  hr.  per  AFHRL  DCR  to  administer  survey  (each 
administration) 

1  hr.  per  AFHRL  DCR  to  conduct  each  interview 

1  hr.  per  AFHRL  DCR  to  write  narrative  from  notes  on 

Interview  Response  Sheet 

1  hr.  Input  of  data  into  AOTS  data  taole  (per  interview) 
1  hr.  per  survey  participant  to  complete  survey 

1  hr.  per  interview  participant  to  respond  to  interview 


3.2.1  Disposition  of  Data.  After  the  surveys  have  been  col¬ 
lected,  the  DCR  will  conduct  a  quick-look  analysis  of  the  OMR 
sheet  to  insure  the  integrity  of  the  data  entered  by  the  respon¬ 
dents.  The  following  plan  will  be  followed: 


-  Verify  that  all  SSANs  are  entered  and  that  the  SSANs 
correspond  with  the  names  and  SSANs  on  the  list  of 
incumbents. 


-  Verify  that  all  response  marks  on  each  OMR  sheet  are 
within  the  tolerance  limits  for  the  optical  scanner. 

-  Verify  that  each  form's  demographic  data  meets  the  stand¬ 
ards  set  on  the  sample  OMR  sheet  for  the  same  sur¬ 
vey. 


If  records  are  found  to  be  out  of  tolerance  or  with  error, 
the  DCR  will  return  the  OMR  sheet  to  the  respondent  for 
immedate  correction. 


If  the  respondent  is  interviewed,  the  DCR  will  perform  the 
quick  look  analysis,  described  above,  before  recording  the 
respondents'  SSAN  and  verbal  responses  on  the  Interview  Response 
Sheet.  ACCURATE  records  must  be  given  to  the  Data  Input 
Representative  (DIR)  for  data  input. 
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PROTOTY PE  APTS  DEPLOYMENT  SURVEY 


DIRECTIONS:  On  the  accompanying  answer  sheet,  please  enter: 

YOUR  NAME  in  the  "NAME  GRID"  block 
TODAY'S  DATE  in  the  "DATE  OF  BIRTH"  block 

YOUR  SSAN  in  columns  3.-9  of  the  "NUMERIC  GRID"  block 
NUMBER  14  in  columns  2  4/2  5  of  the  "NUliERIC  GRID"  block 

Indicate  your  response  to  each  statement  by  filling  in  the  oval, 
on  the  answer  sheet,  that  corresponds  to  the  ] etter  that  repre¬ 
sents  your  opinion. 

Read  every  statement  carefully  and  use  the  following  set  of 
response  options  to  indicate  your  opinion. 

A  B  C  D  E  F  G 

STRONGLY  NEUTRAL  STRONGLY 

DISAGREE  AGREE 


1.  THE  AOTS  MASTER  TASK  LIST  (MTL)  EDITOR  IS  SUITABLE  FOR  USE  IN 
THE  OPERATIONAL  WORKCENTERS. 

2.  THE  AOTS  GENERIC  POSITION  TASK  REQUIREMENTS  (GPTR)  EDITOR  IS 
SUITABLE  FOR  USE  IN  THE  OPERATIONAL  WORKCENTERS. 

3.  THE  AOTS  OPERATIONAL  POSITION  TASK  REQUIREMENTS  (OPTR)  EDITOR 
IS  SUITABLE  FOR  USE  IN  THE  OPERATIONAL  WORKCENTERS. 

4.  THE  AOTS  AIRMAN  TRAINING  RECORDS  ARE  AN  IMPROVEMENT  OVER  THE 
AF  FORM  623. 


5.  THE  AOTS  INDIVIDUAL  TRAINING  REQUIREMENTS  (ITRs)  EDITOR  IS 
SUITABLE  FOR  USE  IN  THE  OPERATIONAL  WORKCENTERS. 

6.  THE  AOTS  PROGRESS  REPORTS  PROVIDE  USEFUL  INFORMATION  IN  THE 
OPERATIONAL  WORKCENTERS . 

?.  THE  AOTS  SCHEDULING  FUNCTION  IS  SUITABLE  FOR  USE  TN  THE 
OPERATIONAL  WORKCENTERS. 

S.  THE  AOTS  PERFORMANCE  TESTS  (ORAL  TEST  GUIDES  &  EVALUATION 
CHECKLISTS)  ARE  SUITABLE  FOR  USE  IN  THE  OPERATIONAL  WORKCENTERS. 

9.  THE  AOTS  KNOWLEDGE  TESTS  ARE  SUITABLE  FOR  USE  IN  THE 
OPERATIONAL  WORKCENTERS. 


.1-44 


PROTOTYPE  AOTS  DEPLOYMENT  SURVEY  Cont. 


Use  the.  following 
opinion. 

set  of 

response 

options 

to  indicate  you 

A  B 

STRONGLY 
DISAGREE 

C 

n 

NEUTRAL 

E 

F  G 

STRONGLY 

AGREE 

10.  THE  AOTS  BEHAVIORAL  OBJECTIVES  ARE  SUITABLE  FOR  USE  IN  THE 
OPERATIONAL  WORKCENTERS . 

11.  THE  AOTS  EQUIPMENT  AND  COMPUTER  PROGRAMS  CAN  BE  USED  IN  OTHER 
OPERATIONAL  WORKCENTERS . 

12.  THE  AOTS  IS  EASY  FOR  THE  TRAINEE  TO  USE. 

13.  THE  AOTS  PROVIDES  REALISTIC  TRAINING  ALTERNATIVES  (CAI,  IVD) 
FOR  WHEN  OPERATIONAL  EQUIPMENT  IS  UNAVAILABLE . 

14.  MORE  TIME  IS  AVAILABLE  FOR  TRAINING  UNDER  AOTS  THAN  UNDER 
CONVENTIONAL  OJT. 

15.  THE  CONDITIONS  FOR  TASK  PROFICIENCY  ARE  MORE  CLEARLY  STATED 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

16.  THE  STANDARDS  FOR  TASK  PROFICIENCY  ARE  MORE  CLEARLY  STATED 
UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

17.  THE  PROTOTYPE  AOTS  encompasses  ALL  PHASES  OF  AIR  FORCE  OJT 
REQUIREMENTS . 

18.  THE  PROTOTYPE  ADVANCED  ON-THE-JOB  TRAINING  SYSTEM  (AOTS)  IS 
SUITABLE  FOR  USE  IN  THE  OPERATIONAL  WORKCENTERS. 


Figure  J . j-i 
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PROTOTYPE  APTS  DEPLOYMENT  INTERVIEW  RESPONSE  PACKET 


RESPONDENT'S  NAMii  &  SSAN 


DATE 


INTERVIEWER'S  NAME  &  SSAN  DATE 

DIRECTIONS  Before  beginning  the  interview,  review  the 
respondent's  Deployment  Survey  OMR  sheet.  Using  the  following 
key,  determine  the  category  in  which  each  response  fits  then 
circle  the  appropriate  response  category  for  each  corresponding 
item  on  this  answer  packet. 

A,  B,  C  =  NEGATIVE 
D  =  NEUTRAL 

E,  F,  G  =  POSITIVE 

Next,  using  a  copy  of  the  Deployment  Survey  as  a  guide,  verbally 
obtain  additional  information  for  any  items  that  the  respondent 
answered  with  a  NEGATIVE  or  POSITIVE  response. 

If  a  response  falls  in  the  NEGATIVE  category,  substitute  the 
words  between  the  parenthesis  with  the  appropriate  terminology 
from  each  item  on  the  survey  and  ask  the  follow  questions: 

a.  WHY  DO  YOU  BELIEVE  THE  (OPTR  EDITOR  IS  NOT  CAPABLE  OF  BEING 
USED  IN  AN  OPERATIONAL  ENVIRONMENT)? 

b.  HOW  COULD  THE  (OPTR  EDITOR  BE  IMPROVED  TO  MAKE  IT  MORE 
FUNCTIONAL  IN  THE  WORKCENTERS ) ? 


If  a  response  fails  in  the  POSITIVE  category,  substitute  the 
words  between  the  parenthesis  with  the  appropriate  terminology 
from  each  item  on  the  survey  and  ask  the  follow  questions: 


a.  WHY  DO  YOU  BELIEVE  THE  (OPTR  EDITOR  WILL  BE  A  SUCCESS  IN  THE 
OPERATIONAL  ENVIRONMENT) ? 


WHAT  DO  YOU  BELIEVE  IS  THE  BEST 


EDITOR) ? 


x:  vjxtiL  \jx 


/r\n  rnn 

X 


Record  the  responses  directly  on  this  interview  response  packet. 
If  more  than  three  lines  are  needed  for  a  response,  use  the  back 
side  of  the  respective  sheet  ensuring  that  all  continued  response 
statements  are  numbered  correctly. 
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RESPONDENT'S  SSAN 


1.  The  response  to  item  1  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 


a. 


lb. 


2.  The  response  to  item  2  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 


2a. 


2b. 


3.  The  response  to  item  3  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 


3a. 


RESPONDENT'S  SSAN 


4.  The  response  to  item  4  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
4a .  _ _ _ _ 


5.  The  response  to  item  5  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
5a  - 


6.  The  response  to  item  6  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
6a . 


RESPONDENT'S  SSAN 


7.  The  response  to  item  7  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 


7a. 


7b. 


8.  The  response  to  item  8  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 


i 


8a . 


8b. 


9.  The  response  to  item  9  was  NEGATIVE  /  NEUTRAL  /  POSITIVE 


RESPONDENT'S  SSAN 


10.  The  response  to  item  10  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
10a. 


11.  The  response  to  item  11  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
11a. 


12.  The  response  to  item  12  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 


12a. 


RESPONDENT'S  SSAN 


13.  The  response  to  item  13  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
13a.  _ 


14.  The  response  to  item  14  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
14a. 


15.  The  response  to  item  15  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 


15a. 


RESPONDENT'S  SSAN 


16.  The  response  to  item  16  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
16a. _ 


16b. 


17.  The  response  to  item  17  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
17a. 


17b. 


18.  The  response  to  item  18  was  NEGATIVE  /  NEUTRAL  /  POSITIVE. 
18a. 


18b. 


Figure  J.3-2 
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J .  4 


OBSERVER  SURVEY 


J.4.1  Data  To  Be  Collected.  The  Observer  Survey  .is  designed  to 
obtain  opinions  from  non-AOTS  participants  on  the  usefulness  of 
various  AOTS  documents  and  functions  that  are  used  to  determine 
training  and  evaluation  requirements.  The  observers  are  asked  to 
evaluate  several  AOTS  materials/products/procedures  and  in  some 
instances,  make  a  comparisons  between  AOTS  and  conventional  OJT 
materials/products . 

Using  a  response  scale  ranging  from  A  to  G  the  respondents  are 
asked  to  indicate  how  strongly  they  disagree  or  agree  with  the 
Statements  listed  on  the  Observer  Survey  (Figure  J.4,  p.57). 
Areas  covered,  with  Appendix  E  references,  are  as  follows: 

— Functions  of  Training  Delivery  Programs  (E-33B) 

— Management  of  Training  in  Workcenters  (E-32A) 

— Capabilities  for  Evaluating  Training  (E-34A) 

— Functions  of  Evaluation  Programs  (E-34B) 

— Training  &  Evaluation  Documents  Comparison  (E-16B) 

— Evaluation  Instruments  &  Procedures  (E-18B) 

— Operational  Equipment  Alternatives  (E-20B) 

— Products  &  Processes  For  Trainer  Effectiveness 
(E-22B) 

J.4. 2  Requirements  for  DCR.  The  DCR  must  be  able  to  repeatedly: 

1.  Plan,  coordinate  and  conduct  a  two-day  AOTS  tour  for 
non-AOTS  participants  representing  various  MAJCOMS, 

2.  Administer  Observer  Surveys. 

J.4. 3  Frequency  cf  Data  Collection.  Data  will  be  collected  one 
time  from  each  MAJCOM  representative  some  time  between  January 
and  April  1989. 

J.4. 4  Data  Collection  Procedure. 

J.4. 4.1  Coordinate  a  two-dav  MAJCOM  observer  tour.  The  DCR 
will : 

1.  Establish/schedule/arrange/conduct  the  following  events: 

a.  AOTS  Briefing 

b.  AOTS  Workcenter  demonstration  and  hands-on  exploration 

c.  Observer  Survey  Administration 

2.  Using  the  records  of  a  single  imaginary  Airman  (to  enable 
the  observers  to  follow  an  airman  through  the  AOTS  procedures) 
assemble  the  following  products  for  the  Observer  Survey  Ad¬ 
ministration  event: 
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a.  AOTS  Airman  Training  Record  (ATR) 

-  History 

-  Individual  Training  Requirements  (ITR) 

b.  Form  623  (Conventional  OJT  record) 

c.  AOTS  Master  Task  List  (MTL) 

d.  AOTS  Generic  Position  Task  Requirements  (GPTR) 

e.  AOTS  Operational  Position  Task  Requirements  (OPTR) 

f.  Job  Qualification  Standard  (JQS)  (Conventional  OJT) 

g.  AOTS  Behavioral  Objective 

h.  AOTS  Off-line  Knowledge  Test 

i.  AOTS  Off-line  Performance  Test 

j.  AOTS  Individual  Position  Qualification  Status  Summary 

l.  AOTS  Unit  Effectiveness  Report 

m.  AOTS  Observer  Surveys  and  Answer  Sheets 

J.4.4.2  Conduct  two-day  MAJCOM  Observer  Tour,  The  DCR  will 
perform  the  following  tasks  during  the  two  day  time  interval: 

1.  Conduct  an  AOTS  briefing  (AM  of  day  one) 

2.  Escort  the  observers  to  the  workcenter  demonstrations 
(day  one) 

3.  Escort  the  observers  to  the  on-line,  hands-on 
exploration  session  in  the  workcenters  or  where  ever 
there  are  available  terminals  (remainder  of  day  one  and 
AM  of  day  two) 

4.  Administer  Observer  Survey  (PM  of  day  two) 
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J.4.5  Administration  of  Surveys.  The  DCR  will  administei 

the  surveys.  The  DCR  will: 

-  Reserve  T-l  conference  room  or  any  class  room  with  a  long 
table. 

-  Obtain  from  DAC  the  appropriate  AOTS  documents  listed 
in  J.4.4.1  #  2 

-  Obtain  a  sufficient  number  of  #  2  pencils,  OMR  sheets 
and  Observer  Surveys  for  distribution  to  the 
scheduled  non-AOTS  participants. 

-  Attend  and  supervise  the  administration  event. 

—  Administer  the  surveys  in  a  room  free  from  distractions 
—  Distribute  the  surveys,  OMR  sheets  and  appropriate 

AOTS  and  conventional  OJT  documents  to  each  participant 
—  Provide  instructions  for  completing  the  surveys 
(see  sample  instructions  below) 

—  Allow  +'he  respondents  to  leave  and  return  to  the 
administration  area  without  the  surveys  (Surveys 
should  remain  with  the  DCR  while  respondents  are  absent 
from  the  snvvey  administration  area) 

—  Answer  any  stions  pertaining  to  the  completion  of 
the  forms 

—  Review  ALL  0-  'heets  before  the  respondents  leave 
the  administrv.-  i  area  and  ensure  that: 

The  correct  Si  N  was  entered 
The  correct  survey  number  was  entered 
All  question  and  statement  responses  were  entered 
—  Collect  all  pencils,  surveys,  OMR  sheets  and 
documents  after  the  surveys  have  been  completed 


SUGGESTIONS  FOR  ADMINISTERING  SURVEYS 

-  Allow  for  errors  (bring  extra  copies  of  surveys  and  OMR 
sheets) 

-  Administer  the  surveys  in  a  conference  room  or  classroom 

-  Apply  a  ratio  of  one  DCR  to  no  more  than  5  respondents 

-  Read  the  following  SAMPLE  INSTRUCTIONS  to  the  respondents 


THE  PURPOSE  OF  THIS  SURVEY  IS  TO  DETERMINE  HOW  WELL  NON-AOTS 
PERSONNEL,  WHO  HAVE  BEEN  INTRODUCED  TO  AOTS  BUT  HAVE  NOT 
OPERATED  THE  AOTS  IN  THEIR  OWN  WORKCENTERS,  ACCEPT  THE  SYSTEM. 
USE  THE  ACCOMPANYING  OMR  SHEET  AND  INDEPENDENTLY  RESPOND  TO  ALL 
QUESTIONS  AND  STATEMENTS.  READ  EACH  STATEMENT  CAREFULLY  BE¬ 


CAUSE  SOME  STATEMENTS  FAVOR  AOTS  AND  SOME  FAVOR  CONVENTIONAL  OJT. 
AFTER  YOU  HAVE  RESPONDED  TO  ALL  OF  THE  STRUCTURED  STATEMENTS,  USE 
THE  BACK  OF  THE  SURVEY  TO  REPORT  ANY  COMMENTS  OR  SUGGESTIONS  THAT 
YOU  HAVE  ABOUT  AOTS.  BE  HONEST  AND  TAKE  AS  LONG  AS  YOU  WANT  TO 
RESPOND  TO  THE  SURVEY. 


J-4.6  Time  Required  for  Data  Collection.  Time  required  for  each 
tour  and  survey  administration  session  is  as  follows: 
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16  hrs.  AFHRL  plan/coordinate  tour 
16  hrs.  Escort  Observers  and  administer  survey 
1  hr.  Input  data  into  AOTS  data  table 
16  hrs.  Per  observer,  plus  travel  time 

J.4.7  Disposition  of  Data.  After  the  surveys  have  been  col¬ 
lected,  the  DCR  will  conduct  a  quick-look  analysis  of  the  OMR 
sheet  to  insure  the  integrity  of  the  data  entered  by  the  respon¬ 
dents.  The  following  plan  will  be  followed: 

Verify  that  all  SSANs  are  entered  and  that  the  SSANs 
correspond  with  the  names  and  SSANs  on  the  list  of 
incumbents. 

-  Verify  that  all  response  marks  on  each  OMR  sheet  are 
within  the  tolerance  limits  for  the  optical  scanner. 

-  Verify  that  each  form's  demographic  data  meets  the  stand¬ 
ards  set  on  the  sample  OMR  sheet  for  the  same  sur¬ 
vey. 

If  records  are  found  to  be  out  of  tolerance  or  with  error, 
the  DCR  will  return  the  OMR  sheet  to  the  respondent  for 
immediate  correction. 


J-56 


OBSERVER  SURVEY 


DIRECTIONS:  On  the  accompanying  answer  sheet,  please  enter; 
your  NAME  in  the  '’NAME  GRID"  block, 

your  SSAN  in  columns  1-9  of  the  "NUMERIC  GRID"  block, 
number  13  in  columns  24/25  of  the  "NUMERIC  GRID"  block. 

Indicate  your  response  to  each  statement  by  filling  in  the  oval, 
on  the  answer  sheet,  that  corresponds  to  the  letter  that 
represents  your  answer  or  opinion. 


1.  WHAT  MAJCOM/SOA  ARE  YOU  REPRESENTING? 


A.  MAC 

F.  ANG 

K.  USAFE 

P.  OTHER 

B.  SAC 

G.  AFCC 

L.  AFLC 

C,  TAC 

H.  AFSC 

M.  SPACECMD 

D.  ATC 

I.  AAC 

N.  ESC 

E.  AFRES 

J.  PACAF 

0.  AFMPC 

WHAT  IS  YOUR  CURRENT  RANK? 

A.  Ab 

I. 

CMSgt 

B. 

Amn 

J. 

CIVILIAN 

C. 

SrA 

K. 

2nd  Lt 

D. 

Sgt 

L. 

1st  Lt 

E. 

SSgt 

M. 

CAPT 

F. 

TSgt 

N. 

MAJ 

G  ♦ 

MSgt 

0. 

LTC 

H. 

SMSgt 

P. 

COL 
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OBSERVER  SURVEY  Cont. 


Use  the  following  set  of  response  options  to  indicate  your 
opinions  towards  the  statements  written  below  and  on  the  next  two 
pages: 


A 

B 

C 

D 

E 

F 

G 

STRONGLY 

NEUTRAL 

STRONGLY 

DISAGREE 

AGREE 

3. 

THE 

AOTS 

IS 

EASY 

TO 

ACCESS. 

4. 

THE 

AOTS 

IS 

EASY 

TO 

USE. 

5.  AOTS  PROVIDES  THE  CAPABILITY  TO  DISPLAY  AN  AIRMAN'S  CURRENT 
TRAINING  STATUS. 

6.  THE  AOTS  SECURITY  SYSTEM  PREVENTS  TRAINEES  FROM  ACCESSING 
UNAUTHORIZED  MENUS,  OPTIONS,  AND  PROCESSES. 

7.  THE  AOTS  SECURITY  SYSTEM  PREVENTS  UNAUTHORIZED  INDIVIDUAL^ 
FROM  ACCESSING  AN  AIRMAN'S  OJT  RECORDS. 

8.  THE  AOTS  SECURITY  SYSTEM  PREVENTS  UNAUTHORIZED  INDIVIDUALS 
FROM  ACCESSING  AOTS  EVALUATION  MATERIALS. 

9.  AOTS  EVALUATION  INSTRUMENTS  (e.g,,  KNOWLEDGE  TEST, 

PERFORMANCE  TEST)  CAN  HELP  SUPERVISORS  IDENTIFY  TASK  TRAINING 
DEFICIENCIES. 

10.  AOTS  EVALUATION  INSTRUMENTS  CAN  HELP  SUPERVISORS  IDENTIFY 
TRAINEES'  TASK  KNOWLEDGE  DEFICIENCIES. 

11.  AOTS  EVALUATION  INSTRUMENTS  CAN  HELP  SUPERVISORS  IDENTIFY 
TRAINEES'  TASK  PERFORMANCE  DEFICIENCIES. 

11.  SUPERVISORS  HAVE  MORE  FLEXIBILITY  FOR  EVALUATING  AIRMEN'S 
TASK  PROFICIENCY  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 

12.  AIRMEN  RECEIVE  MORE  SPECIFIC  TEST  FEEDBACK  UNDER  AOTS  THAN 
UNDER  CONVENTIONAL  OJT. 

13.  SUPERVISORS  CAN  DO  A  BETTER  JOB  OF  DETERMINING  TASK  TRAINING 
REQUIREMENTS  USING  THE  AOTS  OPERATIONAL  POSITION  TASK 
REQUIREMENTS  (OPTR)  PRINTOUTS  THAN  WHEN  USING  THE  TRAINEE'S  JOB 
QUALIFICATION  STANDARD  (JQS) . 


J-58 


OBSERVER  SURVEY  Cont. 


Use  the  following  set  of 
opinions. 

response 

options 

to  indicate  your 

ABC 

D 

E 

F  G 

STRONGLY 

NEUTRAL 

STRONGLY 

DISAGREE 

AGREE 

14.  SUPERVISORS  CAN  DO  A  BETTER  JOB  OF  DETERMINING  AN  AIRMAN'S 
TRAINING  REQUIREMENTS  USING  THE  AOTS  AIRMAN  TRAINING  RECORDS 
(ATR)  WHICH  INCLUDES  TRAINING  HISTORY  AND  THE  INDIVIDUAL 
TRAINING  REQUIREMENTS  (ITR)  THAN  USING  AN  AF  FORM  62  3  (OJT 
RECORD) . 

15.  AIRMAN  TRAINING  RECORDS  ARE  EASIER  TO  MAINTAIN  UNDER  AOTS 
THAN  UNDER  CONVENTIONAL  OJT. 

16.  THE  AOTS  INDIVIDUAL  TRAINING  REQUIREMENTS  (ITR)  ARE  AIRMAN 
SPECIFIC. 

17.  THE  AOTS  ITR  ACCURATELY  IDENTIFIES  AN  AIRMAN'S  CURRENT 
TRAINING  NEEDS. 

18.  AOTS  PROVIDES  STANDARDIZED  PROCEDURES  FOR  EVALUATING  TASK 
PERFORMANCE  IN  AN  OPERATIONAL  SETTING. 

19.  AOTS  PERFORMANCE  TESTS,  WHICH  CONSIST  OF  AN  ORAL  TEST  GUIDE 
AND  A  PERFORMANCE  EVALUATION  CHECKLIST,  ARE  VALID  ASSESSMENTS  OF 
TASK  PROFICIENCY, 

20.  COMPUTER  ASSISTED  INSTRUCTION  (CAI)  LESSONS  DEVELOPED  UNDER 
AOTS  ARE  EASY  TO  USE. 

21.  A  CAI  LESSONS  DEVELOPED  UNDER  AOTS  IS  A  SUITABLE  TRAINING 
ALTERNATIVE  WHEN  OPERATIONAL  EQUIPMENT  IS  UNAVAILABLE. 

22.  THE  PATH  TO  A  CAI  LESSON  THROUGH  THE  AOTS  LOGON  jc  EASY  TO 
FOLLOW. 


23.  INTERACTIVE  VIDEO  DISKS  (IVDs)  DEVELOPED  UNDER  AOTS  ARE  EASY 
TO  USE. 

24.  AN  IVD  DEVELOPED  UNDER  AOTS  IS  A  SUITABLE  TRAINING 
ALTERNATIVE  WHEN  OPERATIONAL  EQUIPMENT  IS  UNAVAILABLE. 
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OBSERVER  SURVEY  Cont. 


Use  the  following  set  of  response  options  to  indicate  your 
opinions. 


ABODE 
STRONGLY  NEUTRAL 

DISAGREE 


STRONGLY 

AGREE 


25.  MORE  PEOPLE  ARE  ABLE  TO  COMPLETE  TRAINING  ON  A  GIVEN  TASK 
WITHIN  THE  SAME  TIME  PERIOD  USING  IVDs  THAN  WHEN  USING 
CONVENTIONAL  OJT. 

26.  TRAINERS  ARE  ABLE  TO  ASSESS  INDIVIDUAL  TRAINING  PROGRESS 
MORE  EFFECTIVELY  UNDER  AOTS  THAN  UNDER  CONVENTIONAL  OJT. 


Figure  J.4 
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J.5  TEST  SUBJECT  DEMOGRAPHIC  DATA 


J.5.1  Data  Collection  Procedure.  Demographic  Data  will  be  col¬ 
lected  for  the  Control  Groups  as  outlined  in  Appendix  H.l.  The 
Control  Groups  for  which  Demographic  Data  will  be  collected  are 
as  follows: 

AFS  431X1  67  AGS — 91st  AMU — Flight  Line 

67  EMS— 45th  &  91st  AMUs — Phase  Dock 
AFS  732X0  MAE  CAMS  Orderly  Room 

67  CSG/DE — Orderly  Room 

J.5.2  Letters  of  Request.  Request  for  DESIRE  Output  Products, 
using  the  letter  shown  in  H.l. 4.1  as  a  model,  will  be  sent  to  the 
following  POC: 


67  CSG/DPM 

J • 5 . 3  Completion  of  Procedure.  Follow  the  completion  of  proce¬ 
dure  found  in  Appendix  H.l. 


J— 6 1 


J.  6 


OJT  RECORDS 


J.6.1  Data  Collection  Procedure. 


J.6.1.1  For  AOTS  workcenters,  derive  data  from  AOTS  prototype 
data  table  by  extracting  Airman  Training  Record  data.  (OJT  data 
will  now  be  computerized  rather  than  being  on  the  Forms  623.) 

DAC  will  prepare  a  request  to  extract  the  data  required  for 
this  analysis.  These  data  include:  name,  SSAN,  rank,  organiza¬ 
tion,  OPTR  duty  position,  duty  title,  time  to  position  qualifica¬ 
tion,  total  number  tasks  required,  percentage  of  tasks  completed, 
by  month. 

J.6.1.2  Comply  with  procedures  found  in  Appendix  H.2  for  the 
following  Control  Groups: 

AFS  431X1  67  AGS— 91st  AMU — Flight  Line 

67  EMS — 45th  &  91st  AMUs — Phase  Dock 
AFS  732X0  MAE  CAMS  Orderly  Room 

67  CSG/DE — Orderly  Room 

J • 6 .2  Letters  of  Request.  Letters  of  request,  indicating  person¬ 
nel  to  be  included  in  the  Control  Groups  will  be  sent:  to  the 
following  POCs: 

67  AGS/MAAAA 
67  CSG/DE A 
67  EMS/MAE 

The  letter  on  the  next  page  may  be  used  as  a  model : 
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SAMPLE  LETTER  NOTIFYING  POCs  OF  CONTROL  GROUPS 
AND  DATA  TO  BE  COLLECTED 


REPLY  TO 

ATTN  OF:  AFHRL/OL-AK 

SUBJECT:  AOTS  Control  Group  and  Data  Requirements 

TO:  67  AGS/MAAA 

1.  The  91st  AMU/Flightline  duty  section  within  your  organization 
have  been  selected  as  part  of  the  control  group  for  the  test  and 
evaluation  of  the  Advanced  On-the-Jofc  Training  System  (AOTS) . 
During  this  time,  members  of  AFHRL  will  need  to  do  the  following: 

a.  Review  and  extract  data  from  AF  Forms  623  to  compare 
with  the  data  coming  from  the  experimental  group. 

b.  Collect  and  review  data  from  the  MDC/Milap  reports. 

c.  Distribute  and  collect  Weekly  Inventory  of  Time  Spent 
(WITS) . 

2.  We  will  work  hand  in  hand  with  you  to  make  these  arrangements 
and  will  do  all  that  we  can  to  be  as  unobtrusive  as  possible. 
Please  feel  free  to  call  me  at  Ext.  2669  if  I  can  be  of  assis¬ 
tance  . 


JACK  L.  BLACKHURST,  Major,  USAF 
Commander,  AFHRL/OL-AK 
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J.6.3  Completion  of  Procedure.  Continue  through  the  completion 
of  procedure  os  lour  1  in  Appendix  H.2  for  Control  Groups. 


J . 7  QUALITY  CONTROL  ASSURANCE  SUMMARIES 


J.7.1  Data  Collection  Procedure.  Continue  Procedure  found  in 
Appendix  H.3,  adding  Control  Groups,  as  follows: 

AFS  431X1  67  AGS — 91st  AMU — Flight  Line 

67  EMS — 45th  &  91st  AMUs— Phase  Dock 

3.1.2  Letters  of  Request.  Letters  of  request,  indicating  per¬ 
sonnel  to  be  included  in  the  Control  Groups  will  be  sent  to  the 
following  POCs: 


67  AGS/MAAAA 
67  EMS/MAE 

The  letter  on  page  75  may  be  used  as  a  model. 

J.7.3  Completion  of  Procedure.  Continue  procedure  found  in 
Appendix  H.3. 
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J.8  SECURITY  POLICE  QUALITY  CONTROL  TREND 

J.8.1  Data  Collection  Procedure.  Continue  Procedure  found  in 
Appendix  H.4. 
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J . 9  AIRCRAFT  MAINTENANCE  DATA 

J-9.1  Data  Collection  Procedure.  Continue  Procedure  found  in 
Appendix  H.7,  adding  Control  Groups,  as  follows: 


AFS  431X1 


67  AGS— 91st  AMU— Flight  Line 
67  EMS — 45th  AMU — Phase  Dock 


J.io  BASE  CRIME  STATISTICS 


J.10.1  Data  Collection  Procedure.  Continue  Procedure 
Appendix  H.6. 


found 


J.ll  WITS  SHEETS 


J-ll-1  Data  Collection  Procedure.  Continue  Procedure  found  in 
Appendix  H.5,  adding  Control  Groups,  as  follows: 

AFS  431X1  67  AGS--91st  AMU— Flight  Line 

67  EMS — 45th  &  91st  AMUs — Phase  Dock 
AFS  732X0  MAE  CAMS  Orderly  Room 

67  CSG/DE — Orderly  Room 

Frequency  of  Data  Collection.  Data  will  be  collected  in 
accordance  with  H.5. 3  except  for  Active  Duty  component  workcen- 
ters  at  Bergstrom  AFB.  These  data  will  now  be  filled  out  and 
collected  monthly  instead  of  biweekly. 

Letters  of  Request.  Letters  of  request,  indicating  per¬ 
sonnel  to  be  included  in  the  Control  Groups  will  be  sent  to  the 
following  POCs: 


67  AGS/MAAAA 
67  CSG/DEA 
67  EMS/MAE 


Completion  of  Procedure.  Continue  through  the  completion 
of  procedure  as  found  in  Appendix  H.5. 
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SECTION  III:  SLT&E  COMPLIANCE  PROCEDURES 


J . 12  EXTERNAL  ENTITIES  INTERFACES 


j.12.1  Data  To  Be  Collected.  Checklists  of  requirements  for  in¬ 
terfaces  (bonds)  between  AOTS  and  external  entities  (Air  Force 
Operational  Measurement  Center,  Publications  Management,  Person¬ 
nel  Center,  SPAS,  CAMS,  HRL,  Personnel  Data  System,  Base  Supply 
System,  CBPO  OJT,  FTD,  Driver's  School)  to  comply  with  the  fol¬ 
lowing  paragraphs  in  the  System  Specification:  3. 1.5. 1.1-2  (pp. 
37-38),  3. 1.5. 1.4-6  (pp.  39-41),  and  3.1.5.1.8-10  (pp.  42-44). 
These  checklists  will  be  completed  by  AFHRL  and  DAC,  using: 


426X2 

System 

Interface 

Checklist- 

-Figure 

J. 12-1, 

P- 

72 

431X1 

System 

Interface 

Checklist- 

-Figure 

J.12-2, 

P- 

73 

732X0 

System 

Interface 

Checklist- 

-Figure 

J.12-3, 

P* 

74 

811XX 

System 

Interface 

Checklist- 

-Figure 

J.12-4, 

P- 

75 

The  participants  will  be  one  member  fromthe  1ST  and  one  member  of 
the  DAC  instructional  technology  team. 

J.12.2  Requirements  for  DCR.  There  are  no  special  requirements 
for  the  DCR.  He/she  needs  only  to  distribute  the  checklists  to 
the  participants  and  answer  procedural  questions  (In  each  case 
where  a  System  Specification  has  been  met,  a  check  mark  will  be 
placed  in  the  blank  preceding  the  item) . 

J.12.3  Frequency  of  Data  Collection.  Data  will  be  collected  one 
time  at  the  end  of  the  first  quarter  of  SLT&E  (early  November 
1988)  unless  there  are  changes  to  the  interfaces  after  the 
completion  of  the  checklists.  Any  such  change  will  necessitate  a 
re-inspection  accompanied  by  completion  of  the  checklist  again. 

J.12.4  Administration  of  Check  Lists.  In  performing  the 
required  procedures  for  administration  of  the  checklists,  the  DCR 
will : 


-  Obtain  copies  of  the  following  AOTS  products:  Master 
Task  List  (MTL)  for  the  AFSCs;  Airman  Training  Record 
(ATR)  for  one  person,  at  random,  from  each  AFS . 

Obtain  copies  of  the  following  Air  Force  documents: 
USAFOMC  Occupational  Survey  Data  for  each  AFSC;  Policy 
Changes  from  AFS  Functional  Manager  at  AFMPC,  printouts 
and/or  rosters  of  Course,  Training,  &  Evaluation  Require¬ 
ments  from  CAMS,  CBPO,  SPAS  and  Units  of  Assignment. 
Obtain  an  appropriate  number  of  copies  of  the  System 
Checklists. 
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Reserve  T-l  Conference  Room  for  administering  the  check¬ 
lists.  (All  participants  will  meet  at  the  same  time,  al¬ 
though  separate  iterations  may  be  held,  if  desired.) 

-  Notify  participants  of  the  time  and  place  to  meet. 

-  Oversee  the  completion  of  the  checklists. 


J.12.5  Time  Required  for  Data  Collection. 


1  wk.  Set  up  inspection  procedure 
1  hr.  OCR  oversee/correlate  inspection 

1  hr.  per  individual — Complete  checklist 


J.12.7  Disposition  of  Data.  After  the  checklists  have  been  col¬ 
lected,  they  will  be  turned  over  to  DAC;  DAC  will  review  t 
checklist  responses  to  insure  that  the  interfaces  are,  in  fac 
present. 
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426X2  SYSTEM  INTERFACE  CHECKLIST 


DIRECTIONS .  Inspect  the  pertinent  AOTS  and  non  AOTS  documents  to 
determine  if  an  interface  (bond)  exists  between  AOTS  and  the  com¬ 
ponent  identified  in  each  item  below.  Write  a  T  on  the  line  to 
the  left  of  each  numbered  item  if  you  believe  the  statement  is 
true.  Write  an  F  if  you  believe  the  statement  is  false.  Write  a 
U  if  you  are  unable  to  determine  if  the  statement  is  true  or 
false. 


1.  OCCUPATIONAL  SURVEY  DATA  HAVE  BEEN ''SECURED  FROM 
THE  UNITED  STATES  AIR  FORCE  OCCUPATIONAL  MEASUREMENT 
CENTER  (USAFOMC)  FOR  AFS  426X2, 

2.  THE  AOTS  AFS  426X2  MASTER  TASK  LIST  (MTL)  CONTAINS 
TASKS  THAT  HAVE  BEEN  TRANSFERRED  DIRECTLY  FROM  THE 
USAFOMC  MTL  (AOTS  ALPHA  00001  THROUGH  10000) . 

3.  THE  AOTS  AFS  426X2  MTL  CONTAINS  TASKS  (NUMBERS  ALPHA 
10001  THROUGH  60000) THAT  ARE  MODIFICATIONS  OF  THE 
USAFOMC  MTL. 

4.  THE  AOTS  AFS  426X2  MTL  CONTAINS  TASKS  (NUMBERS  ALPHA 
60001  AND  HIGHER,  IF  APPROPRIATE)  THAT  HAVE  NOT 

BEEN  INCLUDED  IN  THE  426X2  OCCUPATIONAL  SURVEY  INSTRU¬ 
MENT. 

5.  FOR  THE  PROTOTYPE  AOTS  AFS  426X2,  INTERFACES  WERE  ES¬ 
TABLISHED  TO  ACQUIRE  PRESCRIBED  TRAINING  REQUIREMENTS 
PUBLICATIONS . 

6.  AFMPC  POLICY  CHANGES  IN  AFS  426X2,  IF  ANY,  ARE  INTE¬ 
GRATED  INTO  AOTS. 

7.  AFS  426X2  PERSONNEL  CHANGES  FROM  PERSONNEL  DATA  SYSTEM 
(PDS)  FOR  AIRMEN  IN  AOTS  ARE  UPDATED  IN  THE  AOTS  SYSTEM 
WEEKLY  FOR  ACTIVE  DUTY  AND  MONTHLY  FOR  RESERVES  &  ANG. 

8.  AFS  426X2  COURSE  DATA  IDENTIFIED  BY  THE  CORE  AUTOMATED 
MAINTENANCE  SYSTEM  (CAMS)  ARE  UPDATED  WITHIN  24  HOURS 
OF  RECEIPT. 

9.  AFS  426X2  INTERFACES  HAVE  BEEN  ESTABLISHED  TO  ENABLE 
AOTS  TO  MANUALY  RECEIVE  OFF-LINE  TRAINING  ACTIVITIES 
SCHEDULING  DATA  FROM  NON-AOT5  TRAINING  AGENCIES. 


Figure  J. 12-1 
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431X1  SYSTEM  INTERFACE  CHECKLIST 


DIRECTIONS .  Inspect  the  pertinent  AOTS  and  non  AOTS  documents  to 
determine  if  an  interface  (bond)  exists  between  AOTS  and  the  com¬ 
ponent  identified  in  each  item  below.  Write  a  T  on  the  line  to 
the  left  of  each  numbered  item  if  you  believe  the  statement  is 
true.  Write  an  F  if  you  believe  the  statement  is  false.  Write  a 
U  if  you  are  unable  to  determine  if  the  statement  is  true  or 
false. 


1.  OCCUPATIONAL  SURVEY  DATA  HAS  BEEN  SECURED  FROM  THE 
UNITED  STATES  AIR  FORCE  OCCUPATIONAL  MEASUREMENT  CENTER 
(USAFOMC)  FOR  AFS  431X1. 

2.  THE  AOTS  AFS  431X1  MASTER  TASK  LIST  (MTL)  CONTAINS 
TASKS  (NUMBERS  ALPHA  00001  THROUGH  10000)  THAT  HAVE 
BEEN  TRANSFERRED  DIRECTLY  FROM  THE  USAFOMC  MTL. 

3.  THE  AOTS  AFS  431X1  MTL  CONTAINS  TASKS  (NUMBERS  ALPHA 
10001  THROUGH  60000)  THAT  ARE  MODIFICATIONS  OF  THE 
USAFOMC  MTL  . 

4.  THE  AOTS  AFS  431X1  MTL  CONTAINS  TASKS  (NUMBERS  ALPHA 
60001  AND  HIGHER  IF  APPROPRIATE)  THAT  HAVE  NOT  BEEN 
INCLUDED  IN  THE  431X1  OCCUPATIONAL  SURVEY  INSTRUMENT  . 

5.  FOR  THE  PROTOTYPE  AOTS  AFS  431X1,  INTERFACES  WERE  ES¬ 
TABLISHED  TO  ACQUIRE  PRESCRIBED  TRAINING  REQUIREMENTS 
PUBLICATIONS . 

6.  AFMPC  POLICY  CHANGES  IN  AFS  431X1,  IF  ANY,  ARE  INTE¬ 
GRATED  INTO  AOTS. 

7.  AFS  431X1  PERSONNEL  CHANGES  FROM  PERSONNEL  DATA  SYSTEM 
(PDS)  FOR  AIRMEN  IN  AOTS  ARE  UPDATED  IN  THE  AOTS  SYSTEM 
WEEKLY  FOR  ACTIVE  DUTY  AND  MONTHLY  FOR  RESERVES  &  ANG. 

8.  AFS  431X1  COURSE  DATA  IDENTIFIED  BY  THE  CORE  AUTOMATED 
MAINTENANCE  SYSTEM  (CAMS)  ARE  UPDATED  WITHIN  24  HOURS 
OF  RECEIPT. 

9.  AFS  431X1  INTERFACES  HAVE  BEEN  ESTABLISHED  TO  ENABLE 

AOTS  TO  MANUALLY  RECEIVE  OFF-LINE  TRAINING  ACTIVITIES 
SCHEDULING  DATA  FROM  NCN-AOTS  TRAINING  AGENCIES. 


Figure  J.12-2 
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732X0  SYSTEM  INTERFACE  CHECKLIST 


DIRECTIONS .  Inspect  the  pertinent  AOTS  and  non  AOTS  documents  to 
determine  if  an  interface  (bond)  exists  between  AOTS  and  the  com¬ 
ponent  identified  in  each  item  below.  Write  a  T  on  the  line  to 
the  left  of  each  numbered  item  if  you  believe  the  statement  is 
true.  Write  an  F  if  you  believe  the  statement  is  false.  Write  a 
U  if  you  are  unable  to  determine  if  the  statement  is  true  or 
false. 

_  1.  OCCUPATIONAL  SURVEY  DATA  HAS  BEEN  SECURED  FROM  THE 

UNITED  STATES  AIR  FORCE  OCCUPATIONAL  MEASUREMENT  CENTER 
(USAFOMC)  FOR  AFS  732X0. 

_  2.  THE  AOTS  AFS  732X0  MASTER  TASK  LIST  (MTL)  CONTAINS 

TASKS  (NUMBERS  ALPHA  00001  THROUGH  10000)  THAT  HAVE 
BEEN  TRANSFERRED  DIRECTLY  FROM  THE  USAFOMC  MTL. 

_  3.  THE  AOTS  AFS  732X0  MTL  CONTAINS  TASKS  (NUMBERS  ALPHA 

10001  THROUGH  6G000)  THAT  ARE  MODIFICATIONS  OF  THE 
USAFOMC  MTL. 

_  4.  THE  AOTS  AFS  732X0  MTL  CONTAINS  TASKS  (NUMBERS  ALPHA 

60001  AND  HIGHER,  IF  APPROPRIATE)  THAT  HAVE  NOT 
BEEN  INCLUDED  IN  THE  732X0  OCCUPATIONAL  SURVEY 
INSTRUMENT. 

_  5.  FOR  THE  PROTOTYPE  AOTS  AFS  732X0,  INTERFACES  WERE  ES¬ 
TABLISHED  TO  ACQUIRE  PRESCRIBED  TRAINING  REQUIREMENTS 
PUBLICATIONS . 

_  6.  AFMPC  POLICY  CHANGES  IN  AFS  732X0,  IF  ANY,  ARE  INTE¬ 
GRATED  INTO  AOTS. 

_  7.  AFS  732X0  PERSONNEL  CHANGES  FROM  PERSONNEL  DATA  SYSTEM 

(PDS)  FOR  AIRMEN  IN  AOTS  ARE  UPDATED  IN  THE  AOTS  SYSTEM 
WEEKLY  FOR  ACTIVE  DUTY  AND  MONTHLY  FOR  RESERVES  &  ANG. 

_  8.  AFS  732X0  COURSE  DATA  FOR  PERSONNEL  ASSIGNED  TO  MAIN¬ 
TENANCE  IDENTIFIED  BY  THE  CORE  AUTOMATED  MAINTENANCE 
SYSTEM  (CAMS)  ARE  UPDATED  WITHIN  24  HOURS  OF  RECEIPT. 

_  9-  AFS  732X0  INTERFACES  HAVE  BEEN  ESTABLISHED  TO  ENABLE 

AOTS  TO  MANUALLY  RECEIVE  OFF-LINE  TRAINING  ACTIVITIES 
SCHEDULING  DATA  FROM  NON-AOTS  TRAINING  AGENCIES. 


Figure  J.12-3 
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811XX  SYSTEM  INTERFACE  CHECKLIST 


DIRECTIONS .  Inspect  the  pertinent  AOTS  and  non  AOTS  documents  to 
determine  if  an  interface  (bond)  exists  between  AOTS  and  the  com¬ 
ponent  identified  in  each  item  below.  Write  a  T  on  the  line  to 
the  left  of  each  numbered  item  if  you  believe  the  statement  is 
true.  Write  an  F  if  you  believe  the  statement  is  false.  Write  a 
U  if  you  are  unable  to  determine  if  the  statement  is  true  or 
false. 

_  1.  OCCUPATIONAL  SURVEY  DATA  HAS  BEEN  SECURED  FROM  THE 

UNITED  STATES  AIR  FORCE  OCCUPATIONAL  MEASUREMENT  CENTER 
(USAFOMC)  FOR  AFS  811XX. 

_  2.  THE  AOTS  AFS  811XX  MASTER  TASK  LIST  (MTL)  CONTAINS 

TASKS  (NUMBERS  ALPHA  00001  THROUGH  10000)  THAT  HAVE 
BEEN  TRANSFERRED  DIRECTLY  FROM  THE  USAFOMC  MTL. 

_  3.  THE  AOTS  AFS  811XX  MTL  CONTAINS  TASKS  (NUMBERS  ALPHA 

10001  THROUGH  60000)  THAT  ARE  MODIFICATIONS  OF  THE 
USAFOMC  MTL. 

_  4.  THE  AOTS  AFS  311XX  MTL  CONTAINS  TASKS  (NUMBERS  ALPHA 

60001  AND  HIGHER,  IF  APPROPRIATE)  THAT  HAVE  NOT 
BEEN  INCLUDED  IN  THE  811XX  OCCUPATIONAL  SURVEY 
INSTRUMENT. 

_  5.  FOR  THE  PROTOTYPE  AOTS  AFS  811XX,  INTERFACES  WERE  ES¬ 
TABLISHED  TO  ACQUIRE  PRESCRIBED  TRAINING  REQUIREMENTS 
PUBLICATIONS. 

_  6.  AFMPC  POLICY  CHANGES  IN  AFS  811XX,  IF  ANY,  ARE  INTE¬ 
GRATED  INTO  AOTS. 

_  7.  AFS  811XX  PERSONNEL  CHANGES  FROM  PERSONNEL  DATA  SYSTEM 

(PDS)  FOR  AIRMEN  IN  AOTS  ARE  UPDATED  IN  THE  AOTS  SYSTEM 
WEEKLY  FOR  ACTIVE  DUTY  AND  MONTHLY  FOR  RESERVES  &  ANG. 

_  8-  AFS  811XX  COURSE  DATA  IDENTIFIED  BY  SECURITY  POLICE 

AUTOMATED  SYSTEM  (SPAS)  ARE  UPDATED  WITHIN  24  HOURS  OF 
RECEIPT. 

_  9*  AFS  811XX  INTERFACES  HAVE  BEEN  ESTABLISHED  TO  ENABLE 

AOTS  TO  RECEIVE  OFF-LINE  TRAINING  ACTIVITIES 
SCHEDULING  DATA  FROM  NON-AOTS  TRAINING  AGENCIES. 


Figure  J.12-4 
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J . 13  RESPONSE  PERFORMANCE 


J.13.1  Data  To  Be  Collected.  Terminal  response  times  and  Master 
Task  List  search  time  to  determine  if  response  performance  goals 
are  met  as  listed  in  the  System  Specification,  Section  3.2. 1.1, 
p .  63  . 

J.13.2  Requirements  for  DCR.  DCRs  must  be  able  to  use  a  stop 
watch  correctly  and  have  a  working  knowledge  of  computer  use  and 
AOTS . 

3-13-2  Frequency  of  Data  Collection.  Tests  will  be  conducted 
one  rime  during  the  second  month  of  SLT&E  (September  1988)  . 

J • 13 . 4  Administration  of  Tests.  DAC  and  AFHRL  will  conduct  the 
tests  in  AFHRL  locations  in  Buildings  T-l,  1808,  and  428  at 
Bergstrom  AFB.  Requirements  for  the  tests  will  be: 

Stop  watch 
-  Operating  PC(s) 

J .  1 3  .  4 . 1  Ope  rat  io  n ...  for  Tests.  Tests  will  be  conducted  as 

follows : 

-  AOTS  will  be  operating  under  normal  load.  "Normal  load" 
will  be  whatever  number  of  users  are  on  line  at  the  time 
the  response  performance  test  occurs,  providing  that  said 
test  shall  occur  during  the  daytime  shift  on  a  regular 
work  day  (Monday  through  Friday  that  is  not  a  holiday) . 

"Terminal  Response  Time"  will  be  the  amount  of  time  re¬ 
quired  from  keyboard  key  press  to  display  on  screen. 

-  For  each  test  (Terminal  Response  and  Master  Task  List 
search),  five  responses/searches  will  be  conducted  and  an 
average  taken  for  time  to  complete.  The  average  time  per 
test  will  be  no  greater  than  listed  below. 

Terminal  Response  Time  2.5  seconds 
Master  Task  List  Search  1.0  minute 

—  Completion  times  will  be  listed  on  a  plain  piece  of  paper 
labeled:  • -terminal  Response  Time"  or  "Master  Task  List 

Search  Time." 

J-13-5  Time  Required  for  Data  Collect: on . 

1  hr.  per  tester 
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J*13.6  Disposition  of  Data.  DAC  will  average  the  times  listed 
for  the  tests  and  verify  the  average  against  the  given  standard 
to  verify  compliance  with  the  Response  Performance  Goals  in  the 
System  Specification. 
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J . 14  HARDWARE  RELIABILITY 


J.14.1  Data  To  Be  Collected.  Reliability  of  the  hardware  IAW 
System  Specification  Section  3. 2. 3.1,  p.  71.  Data  about  the 
workcenter  equipment  will  be  collected  during  its  daytime  opera¬ 
tion.  Reliability  will  be  calculated  by  determining  the  failures 
per  hour  and  the  time  between  failures  for  the  following  pieces 
of  equipment:  printer,  terminal,  monitor,  and  computer  system. 
Data  will  be  collected  from  the  following  logs,  with  their  ac¬ 
companying  procedures  for  use: 

AOTS  Maintenance  Action  Report  Log — Figure  J.14-1,  p.  80 
AOTS  Procedure:  Maintenance  Log — Figure  J.14-2,  p.  81 

Reliability  is  essentially  concerned  with  the  Mean  Time  Between 
Failures  (MTBF)  that  reflects,  in  hours,  the  average  time  a  com¬ 
ponent  or  system  will  operate  without  a  failure  or  error. 
Reliability  excludes  any  failure  caused  by  operator  error, 
natural  disasters,  or  sabotage. 

J.14.2  Requirements  for  DCR.  There  are  no  special  requirements 
for  the  DCR.  The  DCR  will  distribute/collect  the  reports  and 
submit  them  to  DAC  for  analysis. 

J.14.3  Frequency  of  Data  Collection.  Data  gathering  will  be  an 
on-going  process.  Failure  Reporting  Forms  will  be  collected 
quarterly  during  the  reporting  period  of  SLT&E  (Aug  88 — May  89). 

J . 14.4  Collection  of  Reliability  Data.  In  performing  the 
required  procedures  for  collection  of  reliability  data  of  the 
hardware/software,  the  DCR  will: 

-  Obtain/distribute  an  appropriate  number  of  copies  of  the 
logs  and  procedures  for  distribution  to  the  field. 

Distribute  the  logs  and  procedures  to  the  AOTS  workcenter 
librarian. 

Collect  a  copy  of  each  completed  log  at  the  end  of  each 
month  of  the  reporting  period. 

-  Turn  completed  logs  over  to  the  DTM 


r 


i 
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J.14.5  Reliability  Criteria. 


J. 14. 5.1  Equipment  Reliability  Goals.  Following  are  reliability 
goals  during  SLT&E: 


PARAMETER 

THRESHOLD 

ESTIMATE 

MTBF 

250  Hours 

400  Hours 

MTBDE  (Mean  Time  Between 
Downing  Event) s 

400  Hours 

TBD 

Mission  Reliability 

99.5% 

96.5% 

Met 

0.5  Hours 

0.4  Hours 

M  max 

2.0  Hours 

2 . o  Hours 

Mptraax 

32.0  Hours 

24.0  Hours 

Ai 

99.5% 

99.7% 

aa 

99.4% 

TBD 

Ao 

99.8% 

TBD 

Service  Life 

10  Years 

+10  Years 

Operational  Life 

Operating 

On/Off  Cycles 

25,000  Hours 

5,000  Hours 

50,000  Hours 
5,000  Hours 

Time  Base  (for  calculations) 

24  Hours 

N/A 

J.14.6  Time  Required  for  Data  Collection. 

.25  hr.  per  individual  filling  out  log 
1.00  hr.  per  month — log  collection  by  DCR 

J.14.7  Disposition _ o_f  Data.  DCR  will  forward  the  completed 
forms  to  the  DTM  for  analysis  at  the  end  of  each  collection 
period  (quarter) . 
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Figure  J. 14-1 
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AOTS  procedures  maintenance  log 

Date:  Sept.  2,  1986 

Ref:  AOTS  0100-2  (Rev. _ )  Aug.  25,  1986 
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Purpose:  The  maintenance  log  will,  provide  a  means  of  knowing  how  many 
items  are  awaiting  maintenance.  The  maintenance  log  is  used 
in  conjunction  with  the  AOTS  MAINTENANCE  ACTION  REPORT  FORK  .(AOTS  0100- 
1)  .  The  maintenance  log  will  provide  log  numbers  for  each  of  the  AOTS 
MAINTENANCE  ACTION  REPORT  FORMS  for  tracking. 


When  to  use:  The  maintenance  log  is  used  to  record  all  equipmenT 
(hardware)  failures.  The  single  point  contact  person  at  each  site 
(buildings  428,  1808,  and  T2)  will  complete  this  form. 


HOW  TO  USE  THE  FORM: 

1.  A  log  number  is  used  to  track  all  maintenance  action  report  forms* 
This  number  will  be  started  at  each  site  as  follows: 

Log  =  A000G01  for  building  428 

Log  #  E00000I  for  building  1808 

Log  s  COOOOG1  for  building  T2 

2.  The  Date,  Time,  ID#  (normally  the  serial  number) ,  Item  Description 
(PC,  printer,  etc.),  and  M/Org  (maintenance  organization)  is  toHen  frort 
the  AOTS  MAINTENANCE  ACTION  REPORT  FORK. 


3« 

opera 
MAI  NT 
entry 
otner 


when  maintenance  is  complete  and  the  item  is  accepted  as 
tional,  the  Date  Fixed,  Tine  and  Action  are  recorded  fro*  the  ACTfS 
£ NANCE  ACTION  REPORT  FORK .  This  constitutes  closing  out  the  log 

for  this  failure.  (Action  indicates  hard  or  so-tb  failures  &7\d  any 
information)  . 


4.  Page  number  is  recorded  on  the  bottom  for  the  log.  AJsO  location 
of  the  log  is  recorded  (building  423,  1808  or  T2). 


ROUTING: 

^ ■  The  maintenance  log  is  maintained-  by  the  sinqle  point  contact  at 
each  site. 

2.  On  a  periodic  bases,  the  log  is  sent  to  MDAC  in  building .;4 28 . 


Figure  J.14-2 


J . 15  HARDWARE  MAINTAINABILITY 


J.15.1  Data  To  Be  Collected.  Determine  if  AOTS  meets  the 
hardware  maintainability  goals  as  outlined  in  System  Specifica¬ 
tion  Section  3. 2. 4. 2,  p.  78.  Maintainability  of  the  equipment 
during  its  normal  operation  will  be  assessed  from  reports  of 
amount  of  time  to  repair  equipment,  both  On-site  and  Off-site, 
using  the: 

AOTS  Maintenance  Action  Report  Form — Figure  J.15-1,  p.  85 

AOTS  Procedure:  AOTS  Maintenance  Action  Report  Form 

(Figure  J.15-2,  p.  86). 

ClayDesta  Problem  Log — Example,  J.15-3,  p.  87 

Vax  Downtime  Log — Example,  J.15-4,  p.  88 

Maintainability  of  hardware  is,  essentially,  the  Mean-Time-To- 
Repair  (MTTR)  a  component  or  system;  MTTR  is  the  average  number 
of  hours  it  takes  to  repair  a  component  or  system  to  full  operat¬ 
ing  condition. 

J.15.2  Requirements  for  DCR.  There  are  no  special  requirements 
for  the  DCR.  The  DCR  will  distribute/collect  the  reports  and 
submit  them  to  DAC  for  analysis. 

J.15.3  Frequency  of  Data  Collection.  Data  gathering  will  be  an 
on-going  process.  Report  Forms  will  be  collected  quarterly 
during  the  reporting  period  of  SLT&E  (Aug  88 — May  89) . 

J.15.4  Administration  of  Maintainability  Measurement.  In  per¬ 
forming  the  required  procedures  for  administration  of  main¬ 
tainability  measurement  of  the  hardware,  the  DCR  will: 

Insure  that  each  workcenter  single  point  contact  person 
has  copies  of  the  forms. 

-  Collect  copies  of  completed  Maintenance  Action  Report 
Forms,  ClayDesta  Problem  Logs,  and  Vax  Downtime  Logs  at 
the  end  of  each  month  of  the  reporting  period. 
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J. 15. 5 


Maintainability  Criteria. 


J. 15.5.1  Maintainability  Goals  for  Hardware.  The  following 
table  indicates  the  maintainability  goals  for  hardware: 


Parameter 


Threshold 


Mean-Time-To-Repair  (MTTR)  on  the  System 
90%  upper  limit  to  repair 

Mean-Time-To-Restore  the  System 
90%  upper  limit  to  restore 

Mean-Time-To-Remove  and  Replace 
90%  upper  limit  to  remove  and  replace 

Direct  maintenance  manhours  per 
equipment  operating  hour 

Response  time  to  site  of  failure 

Principal  Period  of  Maintenance 

Off  line  maintenance  average  MTTR 
90%  upper  limit  to  repair 

Mean  Preventive  Maintenance  Time 

Mean  Maintenance  Time 

Logistic  Delay  Time 

Maximum  Corrective  Maintenance  Time 

Administrative  Delay  Time 

Maintenance  Downtime 

Maintenance  Manhours/month 

Frequency  of  Preventive  Maintenance 


1.5  Manhours 

2 . 5  Manhours 

0.5  Manhours 
1.0  Manhours 

0.4  Manhours 
1.25  Manhours 

0.05  Hours 

4.0  Hours 

16  Hours/day 
7  Days  per  Week 

1.5  Manhours 
6.0  Manhours 

3 . 0  Hours 

3 . 4  Hours 

4.0  Hours  (90%ile) 
8 . 0  Hours 
2.0  Hours 

7.4  Hours 

6.0  hr/mo 

1  time  per  month 
per  workcenter 


.25  hr.  per  individual  filling  out  form/report 
1.00  hr.  per  month — DCR  form/ report  collection 


i 


J.15.7  Disposition  of  Data.  DCR  will  forward  the  completed 
forms  to  THE  DTM  for  analysis  at  the  end  of  each  collection 
period  (quarter) . 
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AOTS  MAINTENANCE  ACTION  REPORT  FORM 


Log  NUMBER: 


DATE:  _ / _ / _  LOCATION: 


NAME: 


TIME: 


TYPE  OF 

EQUIPMENT: 

u 

l) 

HP  Laserjet+ 

u 

[  ) 

IBM 

PC  AT  XT 

u 

(  1 

DEC 

VAX  0600 

0 

S  o : 

r  .i  a  1 

Number : 

Zenith  Z-248  PC  [] 

Color  printer 
Infotron  Comm 
Other: 


Dot  Matrix  Printer 
Data  Tablet  11  X  11"'' 
Data  Tablet  20  X  20" 


n 

n 


Other  ID" : 


Description  ol  iailure:  _ _ _ _ _ _ 

,1 

_ _ _ _ _ i- 


Problem  reported  to  (responsible  for  maintenance):  _ 

Dare:  / _ y_  _  Time: _ ID: _ 

REPAIR  ACTION:  Dare  maintenance  started:  / _ / _  Ti 

P4.C  •  •  i  i  :n  shed:  _ / _ / _  Time:  _ _ 

[:  On-Site  []  Oil -Site  Name:  _ 

•  :  ••iii  i  :  i  i.m  : 


Time  Taken  to  repair:  _  Time  awaiting  parts: 


Is  the  configuration  of  the  item  changed  during  maintenance?  (]  Yes  l)  No 
If  yes,  describe  and  document  revision  level  changes  and  effected  items. 


Acceptance  of  repaired  item  name  and  signature:  _ 

Certified  by  (name  and  signature) :  _ _ _ 

AOTS  0100-1  (Bov.  _)  Aug.  25,  1986  pi<Jur6  J.15-1 
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AOTS  PROCED  REJ  AOTS  MAINTENANCE  ACTION  REPORT  FORM 
Date:  Sept.  2,  1986 

Ref:  AOTS  for  0100-1  (Rev. _ ).Aug.  25,  1986 


purpose:  Provide  a  procedure,  in  using  the  AOTS  MAINTENANCE  ACTION 
REPORT  FORM.  Procedure  for  handling  the  form  is  also  included.  The 
form  is  also  used  to  record  corrective  actions  taken.  The  form  is  also 
used  to  input  data  to  the  reliability/maihtainability  models. 


When  to  use:  The  AOTS  MAINTENANCE  ACTION  REPORT  FORM  is  to  be  used 
anytime  problems  are  encounter  with  AOTS  equipment  (hardware)  items. 
This  includes  failures  of  equipment  or  uncertain  operation  of  the 
equipment. 


HOW  TO  USE  THE  FORM: 

1.  When  a  problem  is  encountered  with  any  AOTS  equipment  (hardware 
such  as  a  Zenith  Z-243  PC,  printer,  communications,  etc.,,  the  AOTS 
MAINTENANCE  ACTION  REPORT  FORK  is  used  to  report  the  problem. 

The  originator  fills  in  the  following  parts:  NAME,  DATE,  LOCATION, 

TIME,  TYPE  OF  EQUIPMENT,  Serial  Number,  Other  IDS  (if  any),  and  a 
description  of  the  failure. 

2.  The  originator  places  their  name  in  the  top  part  labeled:  NAME _ . 

3.  The  report  is  dated  and  time  is  also  indicated  (standard  military 

time  to  be  used).  The  date  is  entered  month,  day,  year.  ( _ / _ / _ ). 

4.  The  location  where  the  equipment  ts  to  be  entered  in  the  ^rea 
marked  "LOCATION: 


5.  The  TYPE  OF  EQUIPMENT  can  be  checked  if  described  or  written  jn 
under  OTHER.  Tne  serial  number  must  be  filled  in  and  if  any  other  type 
of  identification  number  is  also  used,  that  to  be  filled  ir  under  "Other 
IDS  :  _ 


6.  A  description  of  failure  is  required.  If  there  is  a  printout  or 
other  physical  evidence  of  the  failure,  tr.at  should  be  attached  to  the 
form. 


The  following  to  be  filled  out  by  the  single  point  contact  at  each  site 
(building  428,  1808,  and  T2)-. 

7.  "Problem  report  to:"  section  is  filled  out  by  the  single  point 
contact  at  each  site.  This  refers  to  who' will  actually  perform  the 
maintenance  on  the  failed  item.  (Example  could  be  Infotron  for  the 
communication  equipment  or  Hewlett  Packard  for  the  laserjet  printers.) 

8.  The  date  and  time  the  action  is  reported  to  maintenance 
organization  is  also  filled  out  at  this  tine.  (Date  and  Time  spots.) 
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9-,  .  The  "ID*"  indicates  any  special  numbers.  Example  could  be  a  return 
authorization  number  or  a  name  of  an  individual  who  will  handle  this 
action: 

The  next  section  is  filled  out  by  the  person  who  performs  the  actual 
maintenance.  If, '  the  maintenance  action  is  accomplished  by  a  swap  cf 
component (s) ,  the  single  point  contact  person  continues  to  fill  out  this 
section. 

10.  When  the  date  and  time  of  the  maintenance  is'  started  is  recorded  in 
the  date  and  time  parts. 

11.  Date  and  time  the  maintenance  is  finished  is  recorded  in  the 
appropriate  spots. 

12.  Organization  responsible  for  the  maintenance  is  entered  next 

(Organization:  _ J  . 

13.  Corrective  action  is  a  description  of  what  was* required  to  return 
the  failed  item  back  to  serviceable  condition.  Any  parts  that  failed 
will  be  identified  as  well  as  new  parts  used  to  repair  the  item.  The 
back  of  the  form  can  be  used  for  additional  space  to  write  in.  If  an 
assembly,  subassembly,  or  traceable  component  is  changed  and  the 
revision  level  of  the  new  parts  is  different  from  the  original,  the  new 
revision  levels  is  to  be  recorded  for  this  item. 

14.  Time  take  to  repair  is  to  be  filled  out  as  an  estimate  bv  the 
maintenance  person  after  maintenance  is  complete.  Time  awaiting  parts 
is  required  if  any  delays  are  encountered  in  getting  parts.  The  "  tiaje 
delay  is  to  be  expressed  in  hours. 

The  next  section  is  required  only  if  a  change  to  configuration  occurs. 

•*-^*  the  configuration  of  the  item  is  changed,  revision  level  and 

a-fected  Ci  is  to  be  recorded  by  the  single  point  contact  person. 

nccep^ance  section  is  to  be  comoleted  bv  the  sinale  point  contact 
person. 

16.  acceptance  of  the  equipment  as  operational  and  ready  for  use  is 

required  to  close  out  the  maintenance  action. 

17*_.  CerC-i f ication  of  configuration  is  only  required  is  a  chance  to  the 
conr iguration  occurs.  Certification  indicates  that  the  item  is  fully 
operational  and  meets  all  specifications  for  AOTS . 


ROUTING: 


The  person  having  trouble  with  a  hardware  item  fills  out  the -upper 
eve  of  information.  Date,  time  name,  and  location  followed-,  by  failed 
item  and  description  of  the  problem. 


2. 

site. 

(next 


k®  fon?  is  then  taken  to  the  single  point  contact  person  at  each 
he  single  point  contact  person  logs  the  form.  A  log  number 
m  order)  is  also  placed  at  the  top  of  the  maintenance  action 


form. 


.  The  sinqle  point  contact  person  then  determines  the  correct 

maintenance  group  to  repair  the  failed  item.  This  information  is 
written  into  the  form. 

.  Maintenance  person  fills  out  the  next  repair  section  except  when 

the  maintenance  on  the  item  is  accomplished  by  a  swap  or  similar  method. 
id  tie  Wer  case,  the  single  point,  contact  person  fills  out  the  repair 

action  section. 

s  When  repair  efforts  have  been  finished,  the  form  is  given  bach  to 

the  sTntV  point  contact  person.  Th.e  log  is  closec  out  and  a 

determination  of  configuration  change  is  ">ade;  “  f  -hf  ^ 
occurred  then  the  remaining  action  is  determination  if  -he  unit  ~s 

operational  again. 

6.  If  the  item  is  fully  operational,  the  single  point  contact  person 
at  each  site  accepts  the  unit  bach  into  full  operation. 

7.  only  if  a  configuration  change  has  occurred  will  the  operation  cf 
the  unit  require  certification  as  fully  meeting  operational  requirements 
and  specifications.  This  is  done  by  an  authorized  person  j-rom  .  ^ 
configuration  control  system. 

S.  The  single  point  contact  person  will  file  the  finishes  >.orm.. 

c.  on  a  recular  schedule,  all  forms  will  be  collected  by  and  used 

in  the  reliability  and  maintainability  programs.  The  logs  will  also 
collected  on  a  regular  schedule. 


Figure  J. 15-2 


J-88 


J 


CLAYDESTA  PROBLEM  LOG 


WE  CALLED  CALL  RETURNED  BY 

2:00  6  Nov  86  Philip  Lawrence 

2:30  6  Nov  86  Joann  Cruz 
(ClayDesta) 


3:30  7  Nov  86 


Philip  Lawrence 


4:00  7  Nov  86  Joann  Cruz 
(ClayDesta) 


4:00  17  Nov  86 


Philip  Lawrence 


8:50  18  Nov  86  Joann  Cruz 


1:00  18  Nov  86 


Philip  Lawrence 


3:30  13  Nov  86 


4:00  18  Nov  86 


Philip  Lawrence 


Philip  Lawrence 


4:10  18  Nov  86  Joann  Cruz 


RESULTS 

We  reported  Line  Hits 
to  ClayDesta. 

The  line  problems  were 
intermittant,  and  they  could 
not  duplicate  the  problem. 

We  reported  line  hits  to 
ClayDesta. 

ClayDesta  isolated  the  problem 
to  Southwestern  Bell,  and 
they  called  to  report  it. 
Southwestern  Bell  gave  them 
a  ticket  number  of  "50518, 
and  said  that  they  will  need 
1-2  hours  to  fix  the  line. 

We  reported  Line  Hits  to 
ClayDesta 

They  tested  the  line  last 
night,  and  isolated  the 
problem  to  the  Austin 
Southwestern  Bell  office. 

The  problem  has  not  been 
signed  off  yet.  They  will 
test  again  at  11:30  today. 

I  contacted  Jerry  Farias 
(  ClayDesta  )  who  is 
dealing  with  SW  Bell.  He 
said  that  the  line  would 
be  up  in  1  1/2  hrs. 

I  called  Jerry  who  said 
that  they  had  bumped  the 
priority  to  the  4th  level 
at  SW  Bell. 

I  called  Joann  Cruz,  who 
said  that  they  had  bumped 
the  priority  to  the 
division  level  at  SW  Bell. 

Joann  Cruz  said  SW  Bell 
was  replacing  their  low. 
power  repeaters  to  solve 
the  problem.  She  got  an 
estimated  time  of  completion 
of  5:00  from  SW  Bell. 


7:17  19  Nov  86 


Glenn  McBride 


Figure  J.15-3 


Called  ClayDesta  customer 
service  because  the  line  was 
still  down.  They  had  no  record 
of  any  transactions  after 
11:30  17  Nov  86.  They  did 
not  know  the  current  status 
of  the  line. 
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VAX  DOWN  TIME  LOG 


WE  CALLED 
14  Jan  87 

19  Jan  87 

20  Jan  87 

20  Jan  87 

21  Oan  87 

22  Jan  87 

26  Jan  87 

26  Jan  87 

26  Jan  87 

9  Feb  87 


13  Feb  87 


17  Feb 


20  Feb  87 
11  Mar  87 


PROBLEM 


change  to  Vax\VMS  4. 
with  no  warning. 


Memory  upgrade  to  36 


Keg. 


Logins  disabled  on 
the  Vax.  Lupe  was 
working  with  an  Ether 
Net  upgrade. 

Vax  down  for  EtherNet 
Installation  at  about 


RESULTS 

This  cost  1  hour(  8  -  9  )  of  down 
time  for  a  software  change.  The 
down  time  would  have  been  avoided 
if  we  had  any  warning  about  the  change 

This  cost  4  hours (  8  -  12  )  of  down 
time.  If  we  had  any  warning  of  the 
upgrade,  we  could  have  downloaded 
our  work  to  the  IBM-PC's  and  kept 
working. 

Again  we  had  no  warning.  This  only 
cost  some  early  morning  time(  7  -  8  ) 


We  had  some  warning,  and  it  did  not 
cause  us  problems'.  The  system  was  down 
for  1  hour. 


Vax  down  for  EtherNet 
repairs  at  10:04.  We 
had  no  advance 
warning. 


Again  we  had  no  warning.  We  could 
continue  working  on  the  IBM-PC's 
if  we  had  even  10  minutes  warning, 
system  down  for  3  hours. 


Vax  down  for  repairs  We  were  given  a  warning  banner 
at  8:00.  Up  again  before  the  system  went  down,  but 

at  1:50.  nobody  had  time  to  download  their 

work  since  they  were  just  getting 
here  when  it  went  down. 


Vax  down  for  PM  at  They  gave  us  adequate  warning  this 
10:30.  time. 


Vax  back  up  at  2:45.  No  message  indicating  that  the  system 

was  up  has  appeared. 

Vax  down  again  with  Several  people  were  caught  in  the  middle 
no  warning  at  3:00.  of  editing  files. 


One  of  the  links  from  After  switching  the  lines  on  Sat.,  Lupe 

the  Mux  to  the  Vax  and  Lt. imsand  are  both  unavailable. 

is  not  working  after 

the  links  were 

swapped.  Lupe Yzaguirre 

and  Lt  Imsand  are  not 

there  this  week. 


problem  with  port  TXE7  or  associated  mux  hardware  on  the 
Brooks  end.  Lupe  has  been  notified.  Ke  suggested  that  I 
cnange  all  ports  associated  with  the  bad  port  on  that 
DMZ  and  see  if  problems  continue. 

problem  with  port  TXF7  or  associated  mux  hardware  on  the 
Brooks  end..  Lupe  has  been  notified.  The  line  was  switched 
to  another  line  that  was  unassigned. 

power  outage  at  Brook's  from  7:00  to  8:30. 

Vax  down  approx.  20  min.  for  unknown  reasons. 


i 


) 


t 
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Figure  J.15-4 


J . 16  SYSTEM  AVAILABILITY 


J.16.1  Data  To  Be  Collected.  Data  collected  in  procedures  J.14 
and  J . 15  will  be  used  to  calculate  the  measures  of  availability 
of  the  following:  System,  Computer  System  Component,  Terminal 

Component,  Printer  Components,  and  Digitizer  Pad  Component,  as 
found  in  System  Specification  Section  3.2.5,  p.  83. 

Availability  is  essentially  concerned  with  the  percentage  of  the 
time  equipment  is  available  for  use  as  opposed  to  down  time. 

Requirements  for  passing  the  availability  specifications  are  as 
follows: 


Parameter 


Threshold 


System  Wide  Availability  95% 

Computer  System  Component  Availability  96.5% 

Terminal  Component  Availability  98% 

Printer  Components  Availability  98% 

Digitizer  Pad  Component  Availability  99% 


J-16.2  Requirements  for  DCR.  There  are  no  special  requirements 
for  the  DCR.  He/she  will  turn  over  reports  collected  quarterly 
in  procedures  J.14  and  J.15  to  the  DTM. 

J.16.3  Time  Required  for  Data  Collection.  No  additional  time 
will  be  required  for  data  collection. 

J-16.4  Disposition  of  Data.  Upon  receipt  of  forms  for  proce¬ 
dures  indicated,  DAC  will  analyze  the  information  relative  to 
availability  of  equipment  as  opposed  to  downtime. 
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